Bad performance on windows server 2012

[th1agol1ma12]

Hi everyone,

Sorry if I write something wrong.

I recently install a SAS Visual Analytics (7.3).

Follow the configurations machine:

• Virtual (Hiper-V)
• OS: Windows Server 2012 R2 64bits
• CPU: 24x Intel Xeon 2.20 GHZ 2.19GHZ
• RAM: 128 gb

Other relevant information:

- We do not have concurrent access,
- The largest table loaded should be around 2GB.
- Total 4GB of data in other tables
- Local authentication.

Since it was installed Visual Analytics presents a lot of slowness to execute any activity, login, data explorer, report designer, etc. The features of the machine remain with little use, around 15%.
Accessing the SAS Environment Manager, I noticed a big instability in Apache Tomcat. Apparently it was unavailable several times and was consuming a lot of memory (I believe it is limited in some place).
After restarting the server, processes were faster, login, access to reports, but shortly after the performance degraded again.

Attached some prints I took

I read some tips and tuning tips, but I did not find anything that could help me. Do I need to change any default settings? Make more memory available for some feature? How can I do this?

Thank you very much for your attention.

[SASKiwi]

It looks like you are running your SAS metadata, SAS mid-tier, and SAS VA computer server all on the one Windows server - is this correct? We have a very similar setup except our metadata and mid-tier are on separate Windows virtual servers and we use a physical VA server. We get very good performance.

I suggest you consider having a separate metadata and mid-tier servers.

[th1agol1ma12]

Hi SASKiwi,

Tanks for your answer.

I'll consider this option.

[ThomasPalm]

We had a similar issue; We have the largest Windows box you can imagine, yet everything was so slow.

As the box was top of the pop, it was also configured with 4 (!) network connections, talk about redundant... when we disabled these connections and went with only one connection, all of a sudden, everything was as you would expect it to be...

So; check if you have any weird hardware/network connections, that could mess it all up.

[Kurt_Bremser]

---

@ThomasPalm wrote:

We had a similar issue; We have the largest Windows box you can imagine, yet everything was so slow.

As the box was top of the pop, it was also configured with 4 (!) network connections, talk about redundant... when we disabled these connections and went with only one connection, all of a sudden, everything was as you would expect it to be...

So; check if you have any weird hardware/network connections, that could mess it all up.

---

Those 4 network connectors are built into the box for reasons (failover, load-balancing). Having to disable them for proper peformance speaks volumes about the so-called "operating system".

Man, am I glad my SAS server runs on a UNIX. Light-years ahead.

[JuanS_OCS]

Hello @th1agol1ma12,

great question! I always like questions regarding performance.

Let me start with a summary of the most possible reasons I would expect the performance issues might come from:

1. As @ThomasPalm mentioned, about the network connections
2. Memory settings on the SAS sessions
3. Memory settings on the SASServer12_1
4. Memmory settings on the SASServer1_1 (WIP, SASLogon, all the central  and shared apps), therefore it may become a bottleneck
5. Connection settings on SASServer1_1 (same reason as above)
6. The Environment Manager (same as above, by filling up the connections on SASServer1_1)
7. The WIP/EMI collection/archiving not working properly, and, therefore, filling up the EVMDM database, and, in the end, creating timeouts on the SASServer1_1 connections due to the management of large data that is not archived.

Now, I will go through them. I will try to keep it short (so we can go into more detail where needed) but providing the most relevan info.

1. As @ThomasPalm mentioned, about the network connections

Multiple Ips or mixing Ipv4 with Ipv6 addresses might cause confussion, not only to SAS, but to most of the Web Applications in the market.

I agree with @Kurt_Bremser, while one option is to leave only one IP address working and this would work, they are there for some reason, and it has to be considered also the option to keep those addresses. 

The solution to this problem is to manage the Jgroups connections, and the port where the SAS Web Server (or your Reverse Proxy) is listening to, and bind them to one specific IP address, preferable IPv4, but it can be also IPv6.

http://support.sas.com/kb/44/900.html

http://support.sas.com/kb/35/086.html

http://support.sas.com/documentation/cdl/en/bimtag/69826/PDF/default/bimtag.pdf ( page 349 - Configuring the JGroups Bind Address)

http://support.sas.com/kb/52/670.html (Cache locator)

2. Memory settings on the SAS sessions

Whilest I don't think this is your problem, because it would be reflected only when loading data to LASR, or working the Data Builder, or Viewing a report (therefore, only when working with LASR), it is still worthy a mention for other users looking a t performance issues on VA (and, who knows, it might help you as well).

MEMSIZE, SORTSIZE, CPUCOUNT, THREADS and other memory settings on the SAS sessions might impact a lot on the performance while working with the data. And VA has already recommendations to set those settings.  And it might be different for SAS Foundation, LASR, SASApp, Batch Server, Workspace Server, etc, depending on your configuration, data and usage. 

I can give you recommendations about it, but the best recommendations, and supported ones, will come from SAS always, from SAS Technical Support or the SAS Enterprise Excellent Center (EEC).

3. Memory settings on the SASServer12_1

This applies if only your VA applications are lacking performance. You can test if other apps are slow by testing SAS Stored Process web app or SAS Admin.

I think you already read about it, since you mentioned you did some tuning, but here is the the reference for anyone else interested:

https://support.sas.com/documentation/cdl/en/appsrvtuning/69859/PDF/default/appsrvtuning.pdf

Eventually, you might be able to exceed those settings, if you think it is needed. But beware: it might not be supported, I would do it only if the SAS Environment Manager gives hints about it (as long Garbage Collection time or filled Heap Memory), and only with an approval from SAS Technical Support.

4. Memmory settings on the SASServer1_1 (WIP, SASLogon, all the central  and shared apps), therefore it may become a bottleneck

Similar description as above, but this will apply if all your SAS Web applications are slow, because all the SAS Web apps go through the SASServer1_1, therefore It would mean that you have a shared bottleneck between all your Web Apps, and the main shared resource of the Web Apps, is the SASServer1_1, and the SASWebServer, which centralize the SASLogon, the SAS Content Server/WebDAV, and, to keep it short, the Web Infrastructure Platform).

And same document and recommendations as above. I have seen sometimes required to extend Xms and/or Xmx to around 20 GB temporary, but this should happen only under SAS' approval.

5. Connection settings on SASServer1_1 (same reason as above)

While I cannot see the connections made on your SASServer1_1, on your server.xml configuration on SASServer1_1 are some settings regarding the number of open connections, the pool, that are available for connections. The defaults are fine, although on VA installations the "large" configurations are recommended. And with any change on this setting, you will need to update the settings as well for the setenv/wrapper files and to the Web Server. 

The same document as above will give the hints and path to follow,.

6. The Environment Manager (same as above, by filling up the connections on SASServer1_1)

Sometimes, the Environment Manager might have some problems, especially by collecting the data from agents, and if those connections are open for too long on the initial collection, they might fill up the connections and memory from SASServer1_1. 

You might want to test of this is your case by restarting the complete middle tier, but leaving the SAS Environment Manager server down, and see if your performance gets better or not.

If this is your case, let us know, and let SAS Technical Support to know, because then the EVM and or agents require to be fixed.

7. The WIP/EMI collection/archiving not working properly, and, therefore, filling up the EVMDM database, and, in the end, creating timeouts on the SASServer1_1 connections due to the management of large data that is not archived.

Eventually, the SAS Environment Manager Datamart might need some adjustments and maintenance. If you installed a new 7.3 environment, it should be mostly fine (although a periodic check/revision might be required, or to adjust the times for archiving/cleaning might be nrequired in case the usage of your VA system is high), but if you migrated from a previous version, specially from <7.1 whith no hotfixes.

For this, I highly recommend the reading of:

http://support.sas.com/kb/58/599.html

http://support.sas.com/kb/55/778.html

http://blogs.sas.com/content/sgf/2016/01/26/going-visual-analytics-audit-data-archiving/

http://blogs.sas.com/content/sgf/2015/11/09/controlling-the-size-of-the-web-infrastructure-platform-...

http://support.sas.com/documentation/cdl/en/bimtag/69826/HTML/default/viewer.htm#n06skrc2rtwecsn14vi...

I hop[e it might help a bit! Please let us know how you progress on it.

Best regards,

Juan

[JuanS_OCS]

And now, let me be a bit more technical and going to the basics:

You said:

---

@th1agol1ma12 wrote:

 Follow the configurations machine:

• Virtual (Hiper-V)
• OS: Windows Server 2012 R2 64bits
• CPU: 24x Intel Xeon 2.20 GHZ 2.19GHZ
• RAM: 128 gb

Other relevant information:

- We do not have concurrent access,

---

First, I would like to understand "We do not have concurrent access". What does it mean exactly?

Second: I have a feeling that this is not a sizing provided by SAS. Am I right? Or, at least, it is a deviation from what SAS might have proposed you to provision. The reason because I raise this topic is because, FYI:

• SAS always recommends to have VA on physical machines, rather than VMs. For non-distributed versions of VA it is OK to have a VM, although you need to have in consideration some stuff as:
  • A VM with X cores provides performance as physical machine with X cores - 1.5 core (initial estimation)
  • VMs normally have the storage separated from the VM host, which is a decrease on IO performance (not memory)
  • Some VM configurations are really bad for SAS. An example: dinamic memory allocation. SAS VA works ALOT with RAM, SAS server works better with VMs that have a fixed amount of RAM memory.
• SAS recommends, for VA solutions, a core speed of minimum 2.6 Ghz, and an average of at least 3 Ghz. yours is way lower: 2.20 Ghz. Really low. This is a general recommendation for VA 7.1, so you might want an official update from SAS Technical Support.

• I also see an overkill of amount od CPUs/cores allocated/provisioned for 128 GB or RAM. SAS VA is a high-performance system, which allocates all the cores to work with the RAM memory (even if not all the RAM is in use!). To exceed to be lower from the recommendations may cause your VA systems to have poor performance. For your information, and from VA 7.1 non-distributed:
  • For 128 GB of RAM, 8 cores are the recommendation, and my estimatio would be that this sizing would allow you to work with a biggest table up to 16 GB and load the RAM with uncompressed data sets up to 90 GB (total).

---

[th1agol1ma12]

@JuanS_OCSFirst, thank you very much for your excellent responses.

About the summary of de most possible reasons, I believe that the option

1. Memory settings on the SASServer12_1 aplies in my case, only VA applications are lacking performance. Other applications like SAS Stored Process web app or SAS Admin is just fine.

I followed your suggestion to let the SAS environment manager down, the performance improved on login and some activities. However, report viewing remains very slow.

Answering your questions:

First, I would like to understand "We do not have concurrent access". What does it mean exactly?

Means that at the moment there are no more than one user using visual analytics at the same time.

Second: I have a feeling that this is not a sizing provided by SAS. Am I right?

Yes, you are right.

Let me explain.

This is still a testing environment. It was created to be able to sell the tool in the place where I work. Their environment is fully virtualized so they need Visual Analytics to run this way. If they have the need for a physical machine for this purpose, I believe the project does not go forward, which will be very bad.
Not hiring the SAS sizing was a very bad decision that I made clear to them, but now I have nothing to do.

I keep doing some tests, and reading the notes you sent me. I keep you updated.

Thanks again for your help.