Cache Locator Memory Leak? GemFire taking 70% of SASServer1_1 Memory.

[trix4rabbit]

How do I  limit the amount of memory used by Gemfire on SAS_Server1_1 or clear cache data on SASServer1_1 JVM, so I don't have to allocate 24GB of memory for SASServer1_1?

Here are the stuff we have tested and tried

To keep our SAS 9.4 running on Windows 2012 without restarting, we had to set SASServer1_1 JVM -xmx to 24GB.   We were able to verify the issue we are having is the same as described the links below

https://communities.sas.com/t5/SAS-Visual-Analytics/Bad-performance-on-windows-server-2012/td-p/3438...
=> 4. Memmory settings on the SASServer1_1 (WIP, SASLogon, all the central  and shared apps), therefore it may become a bottleneck...And same document and recommendations as above. I have seen sometimes required to extend Xms and/or Xmx to around 20 GB temporary, but this should happen only under SAS' approval.

https://communities.sas.com/t5/Administration-and-Deployment/Tunning-JVM/td-p/460371
=> ...it is recommended by SAS Technical Support to increase a bit the -Xmx value, from 4 GB to even 20 GB in some cases.

We have cleared out SAS_Audit Logs as based on Problem Note 58589 and have manged our WIP database size.  In addition, we have managed the logs as recommended.  However, we are still seeing SASServer1_1 JVM using up most of the 24GB memory allocated.  

We finally decided to do a SASServer1_1 JVM Heap Dump to identify the source of SASServer1_1 JVM.  We followed steps from Problem Note 61370.  We used the SAS Heap Dump Tool to diagnose Heap Dump File when SASServer1_1 JVM was out of memory.  It identified 70% of the JVM Heap was used by Cache Locator / Gemfire.  I have also verified this using other JVM tools like JMap and Eclipse MAT.   We have done this a few times and everytime, Gemfire is what is using up SASServer1_1 JVM Heap.

The exact class identified using 70% (or 14GB) of SASServer1 JVM heap is "com/gemstone/gemfire/internal/cache/..."

Other Observations

I also noticed that JVM on SASServer1_1 does try to actively to perform GC.  After a major GC, the heap released gets filled backup within minutes, which also traced to GemFire filling up SASServer1_1 JVM Heap. 

The SASServer1_1 JVM memory appears to used up all the heap size when we have more mid-tier web servers cache data to sync between members.

Yes, I have enabled SAS environment manager to trace this issue as well.  The SAS environment manager will report Cache Locator JVM status, which is fine at port 41415.  What I am trying to find out is how Gemfire uses JVM Heap allocated to SASServer1_1.

I have also been reading Pivotal Gemfire which Cache Locator is based upon.  There is a way to evict cache data, but this configuration is not available on SASServer1_1.  In the SASServer1_1 I have gone through the gemfire log and knows it creates storage for the webapps running under SASServer1_1. 

I also know I can try to install the server again as described in https://communities.sas.com/t5/Administration-and-Deployment/Server-performance-slow-down/td-p/19884...

Please don't answer : contact support.  I did and they keep on asking me to provide log files after log files without actually giving me an clear answer.  In addition, I been asked to reproduce the error so support can check and verify ! (This is a production system and I don't want to take it down.  I been keeping the server up by adding more memory, limit user connections and restart SASServer1_1 )

[SimonDawson]

I've never come across a situation where the gemfire instances have data that shouldn't be there. Quite a bit of data is stored in there for users to get work done. While it has cache in the name, quite a bit of the data is only available in there. What you are seeing might be totally normal given the workload applied to the machine. What kind of solutions are deployed here how many users login and use this system?

[SASKiwi]

One of the reasons why your Mid-Tier Web Server1_1 is using a lot of memory could be due to non-optimal performance measurement settings. Also are VA users experiencing slow performance while viewing reports? If Server1_1 is memory constrained then I would not be surprised if VA report viewing is slow, with some refreshes taking a minute or two.

Check the Server1_1 log directory. You will find there at least two log files named tmlog*.*. When the server is performing well these logs will be small - a few KB. If these files are large, like 100s of MBs (one of our was > 1TB!) then you know the server could be overloaded doing performance monitoring.

It was interesting to note your experience with SAS Tech Support. This mirrors our own experience with SAS Tech Support trying to solve recent VA poor performance. The reality of getting to the bottom of VA performance problems is it requires a lot of analysis and supplying a lot of logs. In the end SAS Tech Support did indeed come up with the solution - it just took a long time to get there. I really suggest you keep persevering.. 

[trix4rabbit]

Thanks for the suggestion. I will keep an eye on the size of tmlog*.*

Yes, my user has experienced slow performance in VA, but when this happens, all SAS web apps related slows down too.   I also did a lot of tracing and found that  server12_1 was actually doing okay.  The VA being slow from performance measurement was due to WIP on SASServer1_1.   I had to use JConsole, JMap and other tools to find that SAS Environment Manager impacts WIP and PostgreSQL greatly. 

[SASKiwi]

Please let me know about the size of your tmlog*.* files. If they are large then I can give you a config setting that may help.

[trix4rabbit]

The machine was restarted, and after 12 hours, the tmlog*.* file were different on the web servers.

On the first web server, the file went from tmlog0.log to tmlog210452.log, the file size was 30 meg. 

On the second web server, the file went from tmlog0.log to tmlog1068452.log, but file size remained less than 100KB.

Does this mean I need to change my settings? 

[SASKiwi · Accepted solution]

OK, those file sizes look a bit large.

Here is a setting change to Server1_1 that improved performance a lot for us. Here is a link describing this:

http://support.sas.com/documentation/cdl/en/appsrvtuning/69859/HTML/default/viewer.htm#n11d31ckagwav...

The setting from this page is, assuming you are still using the default value of 500 

Modify the <SAS_CONFIG>\Lev1\Web\WebAppServer\SASServer1_1\lib\jta.properties file and add/modify the following line at the bottom of that file: 
com.atomikos.icatch.checkpoint_interval=50

Of course you will have to stop and start the server for the setting to take effect, and it is worthwhile clearing out the LOG directory at the same time.

[JRoman]

Thanks, @SASKiwi, that one little setting you pointed out has made a HUGE improvement in the performance of our SAS web apps.  Before the change, they were painfully sluggish, to the point of near unusability at times; sure enough, those log files you indicated were massive (~2 GBs).  After making the change, the log files are now mere KBs and the web apps are performing so much better!

[SASKiwi]

@JRoman - fantastic! Glad it helped. It took weeks and weeks of diagnostics with SAS Tech Support to nail that fix. Great you could take advantage of it.

[ShelleySessoms]

Hello @trix4rabbit,

I understand that you do not want the answer of "contact tech support" but sometimes that is really the best answer.

You cannot remove components of the SAS architecture: the SAS Cache Locator is a required component of the SAS deployment.

My best suggestion is to engage your SAS support team (be that tech support or other source) for assistance in reviewing your deployment.

Best,

Shelley