SAS VA Performing very slowly

[pedrammobedi]

As of late, the performance of our Visual Analytics have significantly slowed down. This includes logging in, browsing and loading reports. I know there can be a lot of reasons for this. But I do not remember modifying any settings. Could this be an issue with the server computer?

Anyhow, is there a check list to go through to detect the cause?

[JuanS_OCS]

This seems a performance problem focused on the middle tier. I wonder if the number of your users/concurrent users has increased lately, of if the server (specially middle tier) has not been restarted for some time.

1- I would check if the initial sizing pre-requisites of the VA server has changed. If yes, you should contact your SAS representative. If no, please continue with this little list.

2- I would restart the middle tier services (Web Server, Web Application Servers, Cache Locator, JMS, etc) and archive/clean the termporary files such as logs and the /work and /tmp folders fo the Web Application Servers.

3- If this still is not solving your problem, there is a guide to increase the performance of your middle tier:https://support.sas.com/documentation/cdl/en/appsrvtuning/68923/PDF/default/appsrvtuning.pdf

4- You can always contact SAS Technical Support to advise you on the best way to move forward and keep track of your challenges and achievements.

[pedrammobedi]

Thank you!

I will try the first 3 points you mention and come back to you.

[JuanS_OCS]

Perfect, thank you 🙂 Looking forward for the outcome of your checks/

[pedrammobedi]

Hey! The restart process of my webserver is still running and it has taken much longer than expected. It took about 15 minutes before.

Here I give you the size of my logs, temp and work folders.

1. logs: 13.3 GB
2. temp: 31.2 MB
3. work: 111 MB

Do any of these look unusual? I think the logs folder has become quite bulky. But is there a systematic way of cleaning it or should I just empty the text files manually?

[pedrammobedi]

I have restarted all the services and the web app server. Nothing much has changed. I don't honestly dare approaching the settings mentioned in the Tuning pdf you provided but I am almost sure no one has been here for a while to change them.

[LinusH]

I have no specific experience with VA/LASR logs, but 13 GB sounds quite large.

You may need to review your logging settings.

Also check the logs. There can be a lot of new types of warnings/errors that can be relevant in your investigation. I should be quite easy to see if the logging has elevated at the same time as your performance problems started.

[pedrammobedi]

There are many log files. Whch ones in specific should I check?

Also please take a look at the reply I posted about Java(TM) Platform SE Binary. Thanks!

[pedrammobedi]

I see that Java Platform SE Binary is in average consuming 30-40% of my CPU. It's an active process even when nobody is using SAS VA. Could this be a reason?

[JuanS_OCS]

Hi,

1- as @LinusH said well, those big files on logs, even when no user is there and you already cleaned the logs, it seems to indicate that some of your log configurations are on too high detail level: http://support.sas.com/documentation/cdl/en/bimtag/68217/HTML/default/viewer.htm#p0qy7oulxf2ot7n1bca...

You might want to change some levels from debug to error or info, what it was default.

2- can you get the full command line of the executable Java Platform SE Binary that consumes those resources? 

  a) this can help ups to locate and identify the process that might be an issue.

  b) 

3- I believe to understand your VA installation is on Windows, in that case is non-distributed. How is your deployment looking like? All installed in 1 server? Metadata/Compute/Middle-tier separated? 

4- Do you use the AutoLoad functionality or only Queries/Schemas?

5- Is it enabled recently the Performance Reports/ Environment Manager reports/APM/etc?

[userid007]

as @JuanS_OCS mentioned, the critical issue would be the autoload logs under VA<config>Levn\Applications\SASVisualAnalytics\VisualAnalyticsAdministrator

I had to setup a clean up job to remove all the small log files as the number of files were huge. 

perhaps cleanup your logs directory and if you are in VM's check if there is any cpu wait for your VM.

[pedrammobedi]

I have tried this to no avail. 

[JuanS_OCS]

Hi @pedrammobedi, since some days passed from your original question, I feel curious if you could make progress with your performance issues on VA. Any feedback will be welcome!

Best regards,

Juan

[pedrammobedi]

Hi! Thank you for your response. Unfortunately the problem lingers. And I got more help from you people in the community that I received from SAS. They have simply directed me to learning Environment Manager in which I still do not see anything suspicious. All I know is that Java Platform SE binary is taking a lot of CPU even when nobody is using SAS. However, it is still hard to say what is causing it to work that hard. The only thing I see is my SAS VA taking a very long time to load. So any help is still appreciated. 

[JuanS_OCS]

Hi,

sorry to learn that you still have issues there, but let's see if we can do something about it.

On one hand, I would keep open a track with SAS Technical Support and let them know that Environment Manager is not helping this time. It is a good tool, but not in your case for now.

In the other side, since most processes in SAS (metadata, compute and web tiers) are java processes, first we would need more information about which specific SAS process is giving you the high level of CPU usage, in order to filter from the huge amount of options available where the performance problem may came from.

For that, if you server is Lunux, ps -ef | grep java would help.

If your server is Windows, you can go to the Task Manager, and on the details tab, you can add the Command Line column and try to extend it as much as possible to get the full command.

Something else that may help is to know:

- Do you use AutoLoad or Data Builder Queries ?

- How your deployment looks like? How many servers compose the SAS VA application?