08-29-2016 03:45 AM
As of late, the performance of our Visual Analytics have significantly slowed down. This includes logging in, browsing and loading reports. I know there can be a lot of reasons for this. But I do not remember modifying any settings. Could this be an issue with the server computer?
Anyhow, is there a check list to go through to detect the cause?
09-01-2016 08:55 AM
sorry, what i need is the wrapper.conf under /Config/Lev1/Web/WebAppServer/SASServer1_1/conf. You should find some lines with information such as:
wrapper.java.additional.7=-Xmx4096m wrapper.java.additional.8=-Xss256k wrapper.java.additional.9=-Xms1024m wrapper.java.additional.10=-XX:PermSize=768m wrapper.java.additional.11=-XX:MaxPermSize=1280m
How many cores do you have on the server (logical or virtual), which speed and how much RAM memory?
Now, and as heads up of what my advise will be based on:
SAS recommends (by default) on Single Web App Server ( http://support.sas.com/documentation/cdl/en/appsrvtuning/68923/HTML/default/viewer.htm#n1qc1gus8n8a5... ) the following values. But you should extend them until the performance is good again. I have on some systems values of 20480m for Xms and Xmx and 2048 for PermSize and MaxPermSize. You should start with the recommended values, and then start to make them grow. After each change, you will need to restart the SASServer1_1 service (do not forget to clean or achive and clean SASServer1_1/tmp, SASServer1_1/work and SASServer1_1/logs after stoping and before starting the SASserver1_1:
SharedServices JDBC Resource
Tomcat Executor Thread Pool
More things you can do to tune your system:
And those other two, which can be found under: http://support.sas.com/documentation/cdl/en/appsrvtuning/68923/HTML/default/viewer.htm#n11d31ckagwav...
PS. What @userid007 whanted to mention is the SAS Environment Manager metrics: http://support.sas.com/documentation/cdl/en/appsrvtuning/68923/HTML/default/viewer.htm#p1kwvsj8oujet...
08-29-2016 04:16 AM
This seems a performance problem focused on the middle tier. I wonder if the number of your users/concurrent users has increased lately, of if the server (specially middle tier) has not been restarted for some time.
1- I would check if the initial sizing pre-requisites of the VA server has changed. If yes, you should contact your SAS representative. If no, please continue with this little list.
2- I would restart the middle tier services (Web Server, Web Application Servers, Cache Locator, JMS, etc) and archive/clean the termporary files such as logs and the /work and /tmp folders fo the Web Application Servers.
3- If this still is not solving your problem, there is a guide to increase the performance of your middle tier:https://support.sas.com/documentation/cdl/en/appsrvtuning/68923/PDF/default/appsrvtuning.pdf
4- You can always contact SAS Technical Support to advise you on the best way to move forward and keep track of your challenges and achievements.
08-29-2016 07:31 AM
Hey! The restart process of my webserver is still running and it has taken much longer than expected. It took about 15 minutes before.
Here I give you the size of my logs, temp and work folders.
Do any of these look unusual? I think the logs folder has become quite bulky. But is there a systematic way of cleaning it or should I just empty the text files manually?
08-29-2016 08:32 AM - edited 08-29-2016 08:32 AM
I have restarted all the services and the web app server. Nothing much has changed. I don't honestly dare approaching the settings mentioned in the Tuning pdf you provided but I am almost sure no one has been here for a while to change them.
08-29-2016 09:33 AM
I have no specific experience with VA/LASR logs, but 13 GB sounds quite large.
You may need to review your logging settings.
Also check the logs. There can be a lot of new types of warnings/errors that can be relevant in your investigation. I should be quite easy to see if the logging has elevated at the same time as your performance problems started.
08-29-2016 10:56 AM
1- as @LinusH said well, those big files on logs, even when no user is there and you already cleaned the logs, it seems to indicate that some of your log configurations are on too high detail level: http://support.sas.com/documentation/cdl/en/bimtag/68217/HTML/default/viewer.htm#p0qy7oulxf2ot7n1bca...
You might want to change some levels from debug to error or info, what it was default.
2- can you get the full command line of the executable Java Platform SE Binary that consumes those resources?
a) this can help ups to locate and identify the process that might be an issue.
3- I believe to understand your VA installation is on Windows, in that case is non-distributed. How is your deployment looking like? All installed in 1 server? Metadata/Compute/Middle-tier separated?
4- Do you use the AutoLoad functionality or only Queries/Schemas?
5- Is it enabled recently the Performance Reports/ Environment Manager reports/APM/etc?
08-29-2016 08:24 PM
as @JuanS_OCS mentioned, the critical issue would be the autoload logs under VA<config>Levn\Applications\SASVisualAnalytics\VisualAnalyticsAdministrator
I had to setup a clean up job to remove all the small log files as the number of files were huge.
perhaps cleanup your logs directory and if you are in VM's check if there is any cpu wait for your VM.
09-01-2016 05:30 AM
Hi @pedrammobedi, since some days passed from your original question, I feel curious if you could make progress with your performance issues on VA. Any feedback will be welcome!
09-01-2016 05:39 AM
Hi! Thank you for your response. Unfortunately the problem lingers. And I got more help from you people in the community that I received from SAS. They have simply directed me to learning Environment Manager in which I still do not see anything suspicious. All I know is that Java Platform SE binary is taking a lot of CPU even when nobody is using SAS. However, it is still hard to say what is causing it to work that hard. The only thing I see is my SAS VA taking a very long time to load. So any help is still appreciated.
09-01-2016 05:47 AM - edited 09-01-2016 05:48 AM
sorry to learn that you still have issues there, but let's see if we can do something about it.
On one hand, I would keep open a track with SAS Technical Support and let them know that Environment Manager is not helping this time. It is a good tool, but not in your case for now.
In the other side, since most processes in SAS (metadata, compute and web tiers) are java processes, first we would need more information about which specific SAS process is giving you the high level of CPU usage, in order to filter from the huge amount of options available where the performance problem may came from.
For that, if you server is Lunux, ps -ef | grep java would help.
If your server is Windows, you can go to the Task Manager, and on the details tab, you can add the Command Line column and try to extend it as much as possible to get the full command.
Something else that may help is to know:
- Do you use AutoLoad or Data Builder Queries ?
- How your deployment looks like? How many servers compose the SAS VA application?
Need further help from the community? Please ask a new question.