SAS VA Performing very slowly

[pedrammobedi]

Hi. Thanks for you answer.

This is the full command for the Java prcess that takes most of CPU (the line highlighted in blue):

Let me know if you can read it. (righ click and open in new tab)

And your other questions:

• Yes we use AUTOLOAD but there are only a few data sources and they have been there for a long time. 
• There is only one server for SAS VA afaik.

I also got a much better reply from SAS Support so I paste it below:

The JAVA memory use should on the other hand be related to WEB server activities

(I will check with some technical staff on this)

By the way could you check the following folder:

C:\SAS\Config\Lev1\Web\activemq\data - it should only contain 4 objects and the kahadb-folder content should be of current date and reasonable in size.

When I look at the kahaddb folder there is a log file called db-729.log that is about 32 MB. This file is used by Java and I need to stop the services in order to make any changes. Is this an unusual size for the file? Should I try getting a back up and replacing it with an empty one?

[JuanS_OCS]

Hi,

now it is clear to me that the problem is not on autoload, only the web application server, actially only on the SASServer1_1. And I know this problem, happens specially on the SAS Content Server and SAS Logon.

We are getting there.

Some more questions:

- Do you have only SASServer1_1 or also SASServer2_1 and SASServer12_1?

- Do you have enabled the Audit and Performance reports on VA? (So you have an AUDIT_ARTEFACTS table being created and loading into memory)? And you do, how big is this table at this moment?

- could you please share the JVM settings you have on the wrapper.conf on the SASServer1_1? (if you have more than 1_1, I would like to give a look into those).

Thank you in advance!

Looking forward for your feedback

[pedrammobedi]

Hi again,

• There is only one web app in my services and that is SASServer1_1 - WebAppServer,
• I doubt if there are any audits. At least I do not seem to find the audit_visualanalytics.sas7bdat file in my entire SAS folder.
• This is the content of my wrapper.conf located in the sashome folder.

# Java Application
wrapper.java.command=E:\sas\sashome\SASPrivateJavaRuntimeEnvironment\9.4\jre\bin\java.exe

# Java Classpath (include wrapper.jar)  Add class path elements as
# needed starting from 1
wrapper.java.classpath.1=E:\sas\sashome\SASVersionedJarRepository\eclipse\plugins\JavaServiceWrapper_3.2.3\wrapper.jar
wrapper.java.classpath.2=E:\sas\sashome\SASVersionedJarRepository\eclipse\plugins\sas.launcher.jar

# Java Additional Parameters
wrapper.java.additional.1=-Djava.system.class.loader=com.sas.app.AppClassLoader
wrapper.java.additional.2=-Dsas.app.repository.path="E:\sas\sashome\SASVersionedJarRepository\eclipse"

[userid007]

As @JuanS_OCS mentioned you might to focus on the Java heaps etc.

The audit analytics table will be under emdbla folder if you have enabled the load.

You might want to check the total time spent in garbage collection etx. Easiest way is to enable the auto load for va feed on emi frame work and you have default dashboards provided by sas one is for the application monitoring and another is for va usage.

Under application monitoring you might want to focus on app response time.

Best regards,
Sai korrapati

[pedrammobedi]

I am sorry, but could you be a little more specific like @JuanS_OCS? I haven't enabled AUDIT. And my SAS VA is currently too slow for me to start building reports for monitoring. 

Kind regards,

Pedram

[JuanS_OCS · Accepted solution]

Hey there,

sorry, what i need is the wrapper.conf under /Config/Lev1/Web/WebAppServer/SASServer1_1/conf. You should find some lines with information such as:

wrapper.java.additional.7=-Xmx4096m
wrapper.java.additional.8=-Xss256k
wrapper.java.additional.9=-Xms1024m
wrapper.java.additional.10=-XX:PermSize=768m
wrapper.java.additional.11=-XX:MaxPermSize=1280m

How many cores do you have on the server (logical or virtual), which speed and how much RAM memory?

Now, and as heads up of what my advise will be based on:

SAS recommends (by default) on Single Web App Server ( http://support.sas.com/documentation/cdl/en/appsrvtuning/68923/HTML/default/viewer.htm#n1qc1gus8n8a5... ) the following values. But you should extend them until the performance is good again. I have on some systems values of 20480m for Xms and Xmx and 2048 for PermSize and MaxPermSize. You should start with the recommended values, and then start to make them grow. After each change, you will need to restart the SASServer1_1 service (do not forget to clean or achive and clean SASServer1_1/tmp, SASServer1_1/work and SASServer1_1/logs after stoping and before starting the SASserver1_1:

Large

-Xms -Xmx -XX:PermSize -XX:MaxPermSize

8192m 8192m 1664m 1664m

Large	-Dsas.svcs.http.max.total.connections	512
	-Dsas.svcs.http.max.connections	512

SharedServices JDBC Resource 

Large

maxPoolSize

256

Tomcat Executor Thread Pool

Large

maxThreads

1024

More things you can do to tune your system:

Operating System: http://support.sas.com/documentation/cdl/en/appsrvtuning/68923/HTML/default/viewer.htm#n0xfno04p3q3u...

And those other two, which can be found under: http://support.sas.com/documentation/cdl/en/appsrvtuning/68923/HTML/default/viewer.htm#n11d31ckagwav...

PS. What @userid007 whanted to mention is the SAS Environment Manager metrics: http://support.sas.com/documentation/cdl/en/appsrvtuning/68923/HTML/default/viewer.htm#p1kwvsj8oujet...

[pedrammobedi]

I see. I paste the wrapper.conf content below. I'm now going to read your entire message.

#********************************************************************
# Wrapper Properties
#********************************************************************
set.CATALINA_HOME=..\..
set.CATALINA_BASE=..\..

# Java Main class.  This class must implement the WrapperListener interface
#  or guarantee that the WrapperManager class is initialized.  Helper
#  classes are provided to do this for you.  See the Integration section
#  of the documentation for details.
wrapper.java.mainclass=org.tanukisoftware.wrapper.WrapperSimpleApp

# Java Home
set.JAVA_HOME=E:\sas\sashome\SASPrivateJavaRuntimeEnvironment\9.4\jre

# Java Additional Parameters
wrapper.java.additional.1="-Djava.endorsed.dirs=%CATALINA_HOME%\common\endorsed"
wrapper.java.additional.2="-Dcatalina.base=%CATALINA_BASE%"
wrapper.java.additional.3="-Dcatalina.home=%CATALINA_HOME%"
wrapper.java.additional.4="-Djava.io.tmpdir=%CATALINA_BASE%\temp"
wrapper.java.additional.5=-Djava.util.logging.manager=com.springsource.tcserver.serviceability.logging.TcServerLogManager
wrapper.java.additional.6=-Dwrapper.dump.port=-1
wrapper.java.additional.7=-Xmx4096m
wrapper.java.additional.8=-Xss256k
wrapper.java.additional.9=-Xms1024m
wrapper.java.additional.10=-XX:PermSize=768m
wrapper.java.additional.11=-XX:MaxPermSize=1536m
wrapper.java.additional.12=-XX:NewRatio=8
wrapper.java.additional.13=-XX:+UseConcMarkSweepGC
wrapper.java.additional.14=-XX:+UseTLAB
wrapper.java.additional.15=-XX:+DisableExplicitGC
wrapper.java.additional.16=-XX:+CMSClassUnloadingEnabled
wrapper.java.additional.17=-XX:+UseCompressedOops
wrapper.java.additional.18=-XX:ReservedCodeCacheSize=96m
wrapper.java.additional.19=-Djava.awt.headless=true
wrapper.java.additional.20=-Dsun.rmi.dgc.client.gcInterval=3600000
wrapper.java.additional.21=-Dsun.rmi.dgc.server.gcInterval=3600000
wrapper.java.additional.22=-Dcom.sas.services.logging.disableRemoteList=true
wrapper.java.additional.23=-Dcom.sas.services.logging.disableRemoteLogging=true
wrapper.java.additional.24=-Dcom.sas.log.config.ignoreContextClassLoader=true
wrapper.java.additional.25=-Dwebreportstudio.file.cleanup.interval=60
wrapper.java.additional.26=-Dspring.security.strategy=MODE_INHERITABLETHREADLOCAL
wrapper.java.additional.27=-Dcom.sas.log.config.url="file:///E:/sas/config/Lev1/Web/Common/LogConfig"
wrapper.java.additional.28=-Dmulticast_udp_ip_ttl=1
wrapper.java.additional.29=-Djava.net.preferIPv4Stack=false
wrapper.java.additional.30=-Djava.net.preferIPv6Addresses=false
wrapper.java.additional.31=-Dmulticast.address=239.168.27.15
wrapper.java.additional.32=-Dmulticast.port=8561
wrapper.java.additional.33=-Dsas.jms.authentication.decorator=false
wrapper.java.additional.34=-Dsas.scs.host=kssas.alotteri.se
wrapper.java.additional.35=-Dsas.scs.repository.dir="E:/sas/config/Lev1/AppData/SASContentServer/Repository"
wrapper.java.additional.36=-Dcom.sas.server.isclustered=true
wrapper.java.additional.37=-Dsas.scs.cas.host=kssas.alotteri.se
wrapper.java.additional.38=-Dsas.scs.cas.port=80
wrapper.java.additional.39=-Dsas.scs.cas.scheme=http
wrapper.java.additional.40=-Dsas.scs.svc.host=kssas.alotteri.se
wrapper.java.additional.41=-Dsas.scs.svc.port=80
wrapper.java.additional.42=-Dsas.scs.svc.scheme=http
wrapper.java.additional.43=-Dsas.scs.scheme=http
wrapper.java.additional.44=-Dsas.auto.publish.protocol=http
wrapper.java.additional.45=-Dsas.container.identifier=vfabrictcsvr
wrapper.java.additional.46=-Dsas.cache.locators=kssas.alotteri.se[41415]
wrapper.java.additional.47=-Dgemfire.conserve-sockets=false
wrapper.java.additional.48=-Dspring.profiles.active=locators
wrapper.java.additional.49=-Dsas.ttfontsvert.install.dir="E:/sas/sashome/ReportFontsforClients/9.4"
wrapper.java.additional.50=-Dsas.bivaprint.install.dir="E:/sas/sashome/SASVisualAnalyticsPrintingSupport/7.3"
wrapper.java.additional.51=-Dsas.scs.port=8080
wrapper.java.additional.52=-Dnet.sf.ehcache.skipUpdateCheck=true
wrapper.java.additional.53=-Dorg.terracotta.quartz.skipUpdateCheck=true
wrapper.java.additional.54=-Dsas.auto.publish.host=kssas.alotteri.se
wrapper.java.additional.55=-Dsas.auto.publish.port=8080
wrapper.java.additional.56=-Dsas.appserver.instance.id=SASServer1_1_kssas.alotteri.se
wrapper.java.additional.57=-Dconfig.lev.web.appserver.logs.dir="E:/sas/config/Lev1/Web/Logs/SASServer1_1"
wrapper.java.additional.58=-Djava.security.auth.login.config="E:/sas/config/Lev1/Web/WebAppServer/SASServer1_1/conf/jaas.config"
wrapper.java.additional.59=-Dsas.metadata.use.cluster.properties=true
wrapper.java.additional.60=-Dsas.deploy.dir="E:/sas/config/Lev1/Web/WebAppServer/SASServer1_1/sas_webapps"
wrapper.java.additional.61=-Dsas.deployment.agent.client.config="E:/sas/sashome/SASRemoteDeploymentAgentClient/2.1/config/deployagtclt.properties"
wrapper.java.additional.62=-Dsas.app.repository.path="E:/sas/sashome/SASVersionedJarRepository/eclipse"

# Java Library Path
wrapper.java.library.path.1=%CATALINA_BASE%\bin\winx86_64

# Java Classpath
wrapper.java.classpath.1=%CATALINA_BASE%\bin\winx86_64\wrapper.jar
wrapper.java.classpath.2=%CATALINA_BASE%\bin\winx86_64\threaddumpwrapper.jar
wrapper.java.classpath.3=%CATALINA_HOME%\bin\bootstrap.jar
wrapper.java.classpath.4=%CATALINA_HOME%\bin\tomcat-juli.jar
wrapper.java.classpath.5=%CATALINA_BASE%\lib\log4j.jar
wrapper.java.classpath.6=%CATALINA_BASE%\lib
wrapper.java.classpath.7=%CATALINA_BASE%\conf
wrapper.java.classpath.8=%JAVA_HOME%\..\lib\tools.jar

# Java Application
wrapper.java.command=E:\sas\sashome\SASPrivateJavaRuntimeEnvironment\9.4\jre\bin\java.exe

# Application parameters.  Add parameters as needed starting from 1
# The first application parameter is the name of the class whose main
# method is to be called when the application is launched.  The class
# name is followed by the number of parameters to be passed to its main
# method.  Then comes the actual parameters.
#wrapper.app.parameter.1=org.apache.catalina.startup.Bootstrap
wrapper.app.parameter.1=net.covalent.tomcat.wrapper.ThreadDumpWrapper
wrapper.app.parameter.2=start

wrapper.cpu.timeout=5
wrapper.shutdown.timeout=60
wrapper.jvm_exit.timeout=60

wrapper.request_thread_dump_on_failed_jvm_exit=TRUE

#********************************************************************
# Wrapper Logging Properties
#********************************************************************
# Format of output for the console.  (See docs for formats)
wrapper.console.format=PM

# Log Level for console output.  (See docs for log levels)
wrapper.console.loglevel=WARN

# Log file to use for wrapper output logging.
wrapper.logfile=%CATALINA_BASE%\logs\wrapper.log

# Format of output for the log file.  (See docs for formats)
wrapper.logfile.format=LPTM

# Log Level for log file output.  (See docs for log levels)
wrapper.logfile.loglevel=WARN

# Maximum size that the log file will be allowed to grow to before
#  the log is rolled. Size is specified in bytes.  The default value
#  of 0, disables log rolling.  May abbreviate with the 'k' (kb) or
#  'm' (mb) suffix.  For example: 10m = 10 megabytes.
wrapper.logfile.maxsize=10m

# Maximum number of rolled log files which will be allowed before old
#  files are deleted.  The default value of 0 implies no limit.
wrapper.logfile.maxfiles=10

# Log Level for sys/event log output.  (See docs for log levels)
wrapper.syslog.loglevel=NONE

#********************************************************************
# Wrapper Windows Properties
#********************************************************************
# Title to use when running as a console
wrapper.console.title=%CATALINA_BASE%

#********************************************************************
# Wrapper Windows NT/2000/XP Service Properties
#********************************************************************
# WARNING - Do not modify any of these properties when an application
#  using this configuration file has been installed as a service.
#  Please uninstall the service before modifying this section.  The
#  service can then be reinstalled.

# Service dependencies.  Add dependencies as needed starting from 1
wrapper.ntservice.dependency.1=

# Mode in which the service is installed.  AUTO_START or DEMAND_START
wrapper.ntservice.starttype=AUTO_START

# Allow the service to interact with the desktop.
wrapper.ntservice.interactive=false

#wrapper.ntservice.id is set in the tcruntime-ctl.bat script
wrapper.ntservice.name=SAS [config-Lev1] SASServer1_1 - WebAppServer
wrapper.ntservice.displayname=SAS [config-Lev1] SASServer1_1 - WebAppServer
wrapper.ntservice.description=SASServer1_1 WebAppServer at config-Lev1 on port 8080

[JuanS_OCS]

I see. You probably have too low values on your Java Virtual Machine for the SASServer1_1, therefore probably your java process is taking long time just collecting garbage. If you extend the values as SAS recommends probably will have a good start by itself. But it will depend a lot on your hardware capabilities: RAM and CPU/cores.

[pedrammobedi]

I am just a step away from restarting the services. One quick question: Should I archive the entire SASServer1_1/work folder and then clean it just like the temp and the logs folder?

[JuanS_OCS]

No, work and tmp you can clean directly. And logs it is up to you to first archive them, or clean them directly.

And ig you clean/archive those logs, you can do the same for Lev1\Web\Logs

[pedrammobedi]

I can't thank you enough! We are finally back on track.

[JuanS_OCS]

Great! Very glad to read the good news! 🙂

[S_Shah]

We are facing similar issue on SAS Studio on SAS 9.4 M3. For each JS, CSS, PNG etc file have TTFB (Time to First Byte)//Wait time around two seconds. SAS Studio take abotu 8 minutes to load when the cache is cleared. Can you please post the steop you have taken that resolved the issue?

[SASKiwi]

Adding a new problem to an old post isn't a good idea. For starters you are not even using the same software. Creating a new post will give you much better visibility and you can always reference an old post with a link if you think it will be helpful.

[S_Shah]

Point taken. I have opened new post.