high memory used sas processes

[woo]

Hello, we are using sas 9.4 on linux vm on cloud. When we start sas services all different sas private jre processes started and sometime we are running ouf of memory on mid-tier. we are running with total 32gb total mem. Anyway to free up some mem?

output of ps aux cmd with top 5 high mem used processes...

9.3 .../SASPrivateJavaRuntimeEnvironment/9.4/jre/bin/java
6.3 .../SASPrivateJavaRuntimeEnvironment/9.4/jre/bin/java
2.7 .../SASPrivateJavaRuntimeEnvironment/9.4/jre/bin/java
2.7 .../SASPrivateJavaRuntimeEnvironment/9.4/jre/bin/java
1.9 .../SASPrivateJavaRuntimeEnvironment/9.4/jre/bin/java

[SASKiwi]

Do you have SAS Environment Manager available to you? That will give you a much better idea of the memory usage of your mid-tier JVMs. Each JVM server has its own memory settings so you need to know which are the ones using most of the memory.

[SimonDawson]

The JVM does a lot of memory mapped IO. This means the virtual address space has a lot of pages mapped but this doesn't necessarily mean a lot of memory is being used.

You'd only really want to invest time into looking at this if you start to see high page in/out rates in vmstat.

Run `vmstat 1` and watch the si so columns. If this is mostly 0 then you are not under memory pressure.

[JuanS_OCS]

Hello @woo,

mostly everything in SAS is Java based, hence, this output cannot really help us that much. ps -ef would help us with the full Java command, and to identify what exactly are those few processes.

Anyway, as @SimonDawson mentioned: this does not mean a problem happening in memory. The way that Linux works, towards memory optimization, is that total processes will reserve as much memory as possible, to make them quicker. but not necessarily be used.

And Java JVMs work on a way as well, with Heap memory sizes. JVMs will take a minimum and a max memory size. So as long as your JVM won't reach the Heap (max) size, or the GC (garbage collection) take too much time or work too much, you are fine too.

The way to monitor this:

1- for linux processes, as @SimonDawson mentioned, with vmstat, or even with free -m, you will see the RAM, but also the virtual and cache memory usage.

2- for Java JVMs, with SAS Environment Manager, of with debugging/monitoring java processes, JMX, such as jconsole.

All in all:

if your SAS base or SAS Servers have high memory settings  (-MEMSIZE option, e.g)

or

if you have deployed middle tier (web applications) (SASServerN_M)

it could be possible, depending on the solutions installed, and the memory settings as set up, that each of them could take more RAM as you could expect.

And if you need to understand or even optimize resources, an investigation on your deployment would be required.

Hope it helps.

Kind regards,

Juan

[woo]

Thanks a lot for all your comments.

so far i am seeing zero mem available for mid-tier. tried to see how many processes are running with sas and its like 20+. Also memsize was set up max with workspace server .sh script which i removed. below are some of sas processes in short. i monitored vmstat where si and so are consistently zero. 

$ free -tmg
total used free shared buffers cached
Mem: 47 46 0 0 1 27
-/+ buffers/cache: 17 29
Swap: 5 0 5
Total: 53 46 6

$ ps -aef | grep sas | wc -l
23

ps aux 

%cpu/%mem/...

12.9 11.1 9996664 5493328 ?     Sl   Sep14 527:12 /path/SASPrivateJavaRuntimeEnvironment/9.4/jre/bin/java -Dnop -Djava.util.logging.manager=com.springsource.tcserver.servi

3.5 6.1 7533908 3035792 ? Sl Sep14 143:03 /path/SASPrivateJavaRuntimeEnvironment/9.4/jre/bin/java -Dnop -Djava.util.logging.manager=com.springsource.tcserver.servi

0.8 5.7 7541732 2817400 ? Sl Sep14 34:13 /path/SASPrivateJavaRuntimeEnvironment/9.4/jre/bin/java -Dnop -Djava.util.logging.manager=com.springsource.tcserver.servi

$ vmstat
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
1 0 82604 806156 1201540 28837084 0 0 1 22 0 0 3 1 96 0 0

sasid 26069 1 4 Sep14 ? 02:48:36 /path/SASPrivateJavaRuntimeEnvironment/9.4/jre/bin/java -Xms64M -Xmx1G -Djava.util.logging.config.
file=logging.properties -Djava.security.auth.login.config=/pathconfig/Lev1/Web/activemq/conf/login.config -Dcom.sun.management.jmxremote.port=11099
-Dcom.sun.management.jmxremote.password.file=/pathconfig/Lev1/Web/activemq/conf/jmx.password -Dcom.sun.management.jmxremote.access.file=
/pathconfig/Lev1/Web/activemq/conf/jmx.access -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote -Djava.awt.headless=true
-Djava.io.tmpdir=/pathconfig/Lev1/Web/activemq/tmp -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv6Address=false -Dactivemq.classpath=
/pathconfig/Lev1/Web/activemq/conf:/pathconfig/Lev1/Web/activemq/../lib/ -Dactivemq.home=/pathconfig/Lev1/
Web/activemq -Dactivemq.base=/pathconfig/Lev1/Web/activemq -Dactivemq.conf=/pathconfig/Lev1/Web/activemq/conf -Dactivemq.data=
/pathconfig/Lev1/Web/activemq/data -jar /pathconfig/Lev1/Web/activemq/bin/activemq.jar start

sasid 26154 1 0 Sep14 ? 00:12:13 /path/SASPrivateJavaRuntimeEnvironment/9.4/jre/bin/java -Xms128m -Xmx512m -Dgemfire.log-file=/pathconfig/Lev1/Web/gemfire/instances/
ins_41415/gemfire.log -Dgemfire.mcast-port=0 -Djava.net.preferIPv4Stack=false -Djava.net.preferIPv6Addresses=false -Dgemfire.locators=company.com[41415],company.com[41415] -Dgemfire.
license-application-cache=6M0C3-4VW9H-M8J40-0D52F-DTM0H -classpath /pathconfig/Lev1/Web/gemfire/lib/gemfire.jar:/pathconfig/Lev1/Web/gemfire/lib/antlr.jar:/pathconfig/Lev1/Web/gemfire/
lib/gfsh.jar com.gemstone.gemfire.distributed.Locator 41415

sasid 26359 1 0 Sep14 ? 00:00:25 /path/SASWebServer/9.4/httpd-2.2/bin/httpd.worker -d /pathconfig/Lev1/Web/WebServer
sasid 26361 26359 0 Sep14 ? 00:00:00 /path/SASWebServer/9.4/httpd-2.2/bin/rotatelogs -l /pathconfig/Lev1/Web/WebServer/logs/error_%Y-%m-%d-%H.%M.log 50M
sasid 26362 26359 0 Sep14 ? 00:00:04 /path/SASWebServer/9.4/httpd-2.2/bin/rotatelogs -l /pathconfig/Lev1/Web/WebServer/logs/access_%Y-%m-%d-%H.%M.log 50M
sasid 26364 26359 0 Sep14 ? 00:00:46 /path/SASWebServer/9.4/httpd-2.2/bin/httpd.worker -d /pathconfig/Lev1/Web/WebServer
sasid 26365 26359 0 Sep14 ? 00:00:44 /path/SASWebServer/9.4/httpd-2.2/bin/httpd.worker -d /pathconfig/Lev1/Web/WebServer

sasid 26474 1 12 Sep14 ? 08:44:07 /path/SASPrivateJavaRuntimeEnvironment/9.4/jre/bin/java -Dnop -Djava.util.logging.manager=com.springsource.tcserver.serviceability.
logging.TcServerLogManager -Xmx4096m -Xss256k -Xms1024m -XX:PermSize=512m -XX:MaxPermSize=1024m -d64 -XX:NewRatio=8 -XX:+UseConcMarkSweepGC -XX:+UseTLAB -XX:+DisableExplicitGC
-XX:+CMSIncrementalMode -XX:+CMSClassUnloadingEnabled -XX:+UseCompressedOops -XX:ReservedCodeCacheSize=96m -Djava.awt.headless=true -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.
dgc.server.gcInterval=3600000 -Dcom.sas.services.logging.disableRemoteList=true

sasid 29992 1 0 Sep14 ? 00:04:24 /pathconfig/Lev1/Web/SASEnvironmentManager/agent-5.8.0-EE/wrapper/sbin/../../wrapper/sbin/wrapper-linux-x86-64 /pathconfig/Lev1/Web/
SASEnvironmentManager/agent-5.8.0-EE/wrapper/sbin/../../conf/wrapper.conf wrapper.syslog.ident=hq-agent wrapper.pidfile=/pathconfig/Lev1/Web/SASEnvironmentManager/agent-5.8.0-EE/
wrapper/sbin/../../wrapper/hq-agent.pid wrapper.name=hq-agent wrapper.displayname=HQ Agent wrapper.daemonize=TRUE

sasid 29997 29992 4 Sep14 ? 02:59:27 /path/SASPrivateJavaRuntimeEnvironment/9.4/jre/bin/java -Djava.security.auth.login.config=../../bundles/agent-5.8.0/jaas.config -Xmx128m
-Dagent.install.home=../.. -Dagent.bundle.home=../../bundles/agent-5.8.0 -Dsun.net.inetaddr.ttl=60 -XX:+UseConcMarkSweepGC -Dhttps.protocols=TLSv1.2 -Djdk.tls.client.protocols=TLSv1.2
-Xmx256m -Djava.library.path=%LD_LIBRARY_PATH%:../../wrapper/lib

[JuanS_OCS]

@woo,

I see the processes have data of 14 September, Did you restarted services and ensured no sas processes run before restarting services, since you made the configuration changes? 

Something else: For the process that consumes 12.9 11.1 of CPU and RAM, could you please get the full command line?