Solved: Re: SAS VA Performing very slowly

pedrammobedi · Posted 08-29-2016 03:45 AM

As of late, the performance of our Visual Analytics have significantly slowed down. This includes logging in, browsing and loading reports. I know there can be a lot of reasons for this. But I do not remember modifying any settings. Could this be an issue with the server computer?

Anyhow, is there a check list to go through to detect the cause?

JuanS_OCS · Posted 09-01-2016 08:55 AM

Hey there,

sorry, what i need is the wrapper.conf under /Config/Lev1/Web/WebAppServer/SASServer1_1/conf. You should find some lines with information such as:

wrapper.java.additional.7=-Xmx4096m
wrapper.java.additional.8=-Xss256k
wrapper.java.additional.9=-Xms1024m
wrapper.java.additional.10=-XX:PermSize=768m
wrapper.java.additional.11=-XX:MaxPermSize=1280m

How many cores do you have on the server (logical or virtual), which speed and how much RAM memory?

Now, and as heads up of what my advise will be based on:

SAS recommends (by default) on Single Web App Server ( http://support.sas.com/documentation/cdl/en/appsrvtuning/68923/HTML/default/viewer.htm#n1qc1gus8n8a5... ) the following values. But you should extend them until the performance is good again. I have on some systems values of 20480m for Xms and Xmx and 2048 for PermSize and MaxPermSize. You should start with the recommended values, and then start to make them grow. After each change, you will need to restart the SASServer1_1 service (do not forget to clean or achive and clean SASServer1_1/tmp, SASServer1_1/work and SASServer1_1/logs after stoping and before starting the SASserver1_1:

Large

-Xms

-Xmx

-XX:PermSize

-XX:MaxPermSize

8192m

1664m

Large	-Dsas.svcs.http.max.total.connections	`512`
Large	-Dsas.svcs.http.max.connections	`512`

SharedServices JDBC Resource

Large

maxPoolSize

256

Tomcat Executor Thread Pool

Large

maxThreads

1024

More things you can do to tune your system:

Operating System: http://support.sas.com/documentation/cdl/en/appsrvtuning/68923/HTML/default/viewer.htm#n0xfno04p3q3u...

And those other two, which can be found under: http://support.sas.com/documentation/cdl/en/appsrvtuning/68923/HTML/default/viewer.htm#n11d31ckagwav...

PS. What @userid007 whanted to mention is the SAS Environment Manager metrics: http://support.sas.com/documentation/cdl/en/appsrvtuning/68923/HTML/default/viewer.htm#p1kwvsj8oujet...

View solution in original post

JuanS_OCS · Posted 08-29-2016 04:16 AM

This seems a performance problem focused on the middle tier. I wonder if the number of your users/concurrent users has increased lately, of if the server (specially middle tier) has not been restarted for some time.

1- I would check if the initial sizing pre-requisites of the VA server has changed. If yes, you should contact your SAS representative. If no, please continue with this little list.

2- I would restart the middle tier services (Web Server, Web Application Servers, Cache Locator, JMS, etc) and archive/clean the termporary files such as logs and the /work and /tmp folders fo the Web Application Servers.

3- If this still is not solving your problem, there is a guide to increase the performance of your middle tier:https://support.sas.com/documentation/cdl/en/appsrvtuning/68923/PDF/default/appsrvtuning.pdf

4- You can always contact SAS Technical Support to advise you on the best way to move forward and keep track of your challenges and achievements.

pedrammobedi · Posted 08-29-2016 04:20 AM

Thank you!

I will try the first 3 points you mention and come back to you.

JuanS_OCS · Posted 08-29-2016 04:22 AM

Perfect, thank you 🙂 Looking forward for the outcome of your checks/

pedrammobedi · Posted 08-29-2016 07:31 AM

Hey! The restart process of my webserver is still running and it has taken much longer than expected. It took about 15 minutes before.

Here I give you the size of my logs, temp and work folders.

logs: 13.3 GB
temp: 31.2 MB
work: 111 MB

Do any of these look unusual? I think the logs folder has become quite bulky. But is there a systematic way of cleaning it or should I just empty the text files manually?

pedrammobedi · Posted 08-29-2016 08:32 AM

I have restarted all the services and the web app server. Nothing much has changed. I don't honestly dare approaching the settings mentioned in the Tuning pdf you provided but I am almost sure no one has been here for a while to change them.

LinusH · Posted 08-29-2016 09:33 AM

I have no specific experience with VA/LASR logs, but 13 GB sounds quite large.

You may need to review your logging settings.

Also check the logs. There can be a lot of new types of warnings/errors that can be relevant in your investigation. I should be quite easy to see if the logging has elevated at the same time as your performance problems started.

Data never sleeps

pedrammobedi · Posted 08-29-2016 09:41 AM

There are many log files. Whch ones in specific should I check?

Also please take a look at the reply I posted about Java(TM) Platform SE Binary. Thanks!

pedrammobedi · Posted 08-29-2016 09:26 AM

I see that Java Platform SE Binary is in average consuming 30-40% of my CPU. It's an active process even when nobody is using SAS VA. Could this be a reason?

JuanS_OCS · Posted 08-29-2016 10:56 AM

Hi,

1- as @LinusH said well, those big files on logs, even when no user is there and you already cleaned the logs, it seems to indicate that some of your log configurations are on too high detail level: http://support.sas.com/documentation/cdl/en/bimtag/68217/HTML/default/viewer.htm#p0qy7oulxf2ot7n1bca...

You might want to change some levels from debug to error or info, what it was default.

2- can you get the full command line of the executable Java Platform SE Binary that consumes those resources?

a) this can help ups to locate and identify the process that might be an issue.

b)

3- I believe to understand your VA installation is on Windows, in that case is non-distributed. How is your deployment looking like? All installed in 1 server? Metadata/Compute/Middle-tier separated?

4- Do you use the AutoLoad functionality or only Queries/Schemas?

5- Is it enabled recently the Performance Reports/ Environment Manager reports/APM/etc?

userid007 · Posted 08-29-2016 08:24 PM

as @JuanS_OCS mentioned, the critical issue would be the autoload logs under VA<config>Levn\Applications\SASVisualAnalytics\VisualAnalyticsAdministrator

I had to setup a clean up job to remove all the small log files as the number of files were huge.

perhaps cleanup your logs directory and if you are in VM's check if there is any cpu wait for your VM.

Sai Korrapati

pedrammobedi · Posted 09-01-2016 05:34 AM

I have tried this to no avail.

JuanS_OCS · Posted 09-01-2016 05:30 AM

Hi @pedrammobedi, since some days passed from your original question, I feel curious if you could make progress with your performance issues on VA. Any feedback will be welcome!

Best regards,

Juan

pedrammobedi · Posted 09-01-2016 05:39 AM

Hi! Thank you for your response. Unfortunately the problem lingers. And I got more help from you people in the community that I received from SAS. They have simply directed me to learning Environment Manager in which I still do not see anything suspicious. All I know is that Java Platform SE binary is taking a lot of CPU even when nobody is using SAS. However, it is still hard to say what is causing it to work that hard. The only thing I see is my SAS VA taking a very long time to load. So any help is still appreciated.

JuanS_OCS · Posted 09-01-2016 05:47 AM

Hi,

sorry to learn that you still have issues there, but let's see if we can do something about it.

On one hand, I would keep open a track with SAS Technical Support and let them know that Environment Manager is not helping this time. It is a good tool, but not in your case for now.

In the other side, since most processes in SAS (metadata, compute and web tiers) are java processes, first we would need more information about which specific SAS process is giving you the high level of CPU usage, in order to filter from the huge amount of options available where the performance problem may came from.

For that, if you server is Lunux, ps -ef | grep java would help.

If your server is Windows, you can go to the Task Manager, and on the details tab, you can add the Command Line column and try to extend it as much as possible to get the full command.

Something else that may help is to know:

- Do you use AutoLoad or Data Builder Queries ?

- How your deployment looks like? How many servers compose the SAS VA application?

SAS Innovate 2025: Save the Date