How do I limit the amount of memory used by Gemfire on SAS_Server1_1 or clear cache data on SASServer1_1 JVM, so I don't have to allocate 24GB of memory for SASServer1_1? Here are the stuff we have tested and tried To keep our SAS 9.4 running on Windows 2012 without restarting, we had to set SASServer1_1 JVM -xmx to 24GB. We were able to verify the issue we are having is the same as described the links below https://communities.sas.com/t5/SAS-Visual-Analytics/Bad-performance-on-windows-server-2012/td-p/343837 => 4. Memmory settings on the SASServer1_1 (WIP, SASLogon, all the central and shared apps), therefore it may become a bottleneck...And same document and recommendations as above. I have seen sometimes required to extend Xms and/or Xmx to around 20 GB temporary, but this should happen only under SAS' approval. https://communities.sas.com/t5/Administration-and-Deployment/Tunning-JVM/td-p/460371 => ...it is recommended by SAS Technical Support to increase a bit the -Xmx value, from 4 GB to even 20 GB in some cases. We have cleared out SAS_Audit Logs as based on Problem Note 58589 and have manged our WIP database size. In addition, we have managed the logs as recommended. However, we are still seeing SASServer1_1 JVM using up most of the 24GB memory allocated. We finally decided to do a SASServer1_1 JVM Heap Dump to identify the source of SASServer1_1 JVM. We followed steps from Problem Note 61370. We used the SAS Heap Dump Tool to diagnose Heap Dump File when SASServer1_1 JVM was out of memory. It identified 70% of the JVM Heap was used by Cache Locator / Gemfire. I have also verified this using other JVM tools like JMap and Eclipse MAT. We have done this a few times and everytime, Gemfire is what is using up SASServer1_1 JVM Heap. The exact class identified using 70% (or 14GB) of SASServer1 JVM heap is "com/gemstone/gemfire/internal/cache/..." Other Observations I also noticed that JVM on SASServer1_1 does try to actively to perform GC. After a major GC, the heap released gets filled backup within minutes, which also traced to GemFire filling up SASServer1_1 JVM Heap. The SASServer1_1 JVM memory appears to used up all the heap size when we have more mid-tier web servers cache data to sync between members. Yes, I have enabled SAS environment manager to trace this issue as well. The SAS environment manager will report Cache Locator JVM status, which is fine at port 41415. What I am trying to find out is how Gemfire uses JVM Heap allocated to SASServer1_1. I have also been reading Pivotal Gemfire which Cache Locator is based upon. There is a way to evict cache data, but this configuration is not available on SASServer1_1. In the SASServer1_1 I have gone through the gemfire log and knows it creates storage for the webapps running under SASServer1_1. I also know I can try to install the server again as described in https://communities.sas.com/t5/Administration-and-Deployment/Server-performance-slow-down/td-p/198842 Please don't answer : contact support. I did and they keep on asking me to provide log files after log files without actually giving me an clear answer. In addition, I been asked to reproduce the error so support can check and verify ! (This is a production system and I don't want to take it down. I been keeping the server up by adding more memory, limit user connections and restart SASServer1_1 )
... View more
