topic Re: SAS VA Performing very slowly in SAS Visual Analytics

SAS VA Performing very slowly

pedrammobedi — Mon, 29 Aug 2016 07:45:04 GMT

As of late, the performance of our Visual Analytics have significantly slowed down. This includes logging in, browsing and loading reports. I know there can be a lot of reasons for this. But I do not remember modifying any settings. Could this be an issue with the server computer?

Anyhow, is there a check list to go through to detect the cause?

Re: SAS VA Performing very slowly

JuanS_OCS — Mon, 29 Aug 2016 08:16:31 GMT

This seems a performance problem focused on the middle tier. I wonder if the number of your users/concurrent users has increased lately, of if the server (specially middle tier) has not been restarted for some time.

1- I would check if the initial sizing pre-requisites of the VA server has changed. If yes, you should contact your SAS representative. If no, please continue with this little list.

2- I would restart the middle tier services (Web Server, Web Application Servers, Cache Locator, JMS, etc) and archive/clean the termporary files such as logs and the /work and /tmp folders fo the Web Application Servers.

3- If this still is not solving your problem, there is a guide to increase the performance of your middle tier:https://support.sas.com/documentation/cdl/en/appsrvtuning/68923/PDF/default/appsrvtuning.pdf

4- You can always contact SAS Technical Support to advise you on the best way to move forward and keep track of your challenges and achievements.

Re: SAS VA Performing very slowly

pedrammobedi — Mon, 29 Aug 2016 08:20:40 GMT

Thank you!

I will try the first 3 points you mention and come back to you.

Re: SAS VA Performing very slowly

JuanS_OCS — Mon, 29 Aug 2016 08:22:58 GMT

Perfect, thank you 🙂 Looking forward for the outcome of your checks/

Re: SAS VA Performing very slowly

pedrammobedi — Mon, 29 Aug 2016 11:31:15 GMT

Hey! The restart process of my webserver is still running and it has taken much longer than expected. It took about 15 minutes before.

Here I give you the size of my logs, temp and work folders.

logs: 13.3 GB
temp: 31.2 MB
work: 111 MB

Do any of these look unusual? I think the logs folder has become quite bulky. But is there a systematic way of cleaning it or should I just empty the text files manually?

Re: SAS VA Performing very slowly

pedrammobedi — Mon, 29 Aug 2016 12:32:58 GMT

I have restarted all the services and the web app server. Nothing much has changed. I don't honestly dare approaching the settings mentioned in the Tuning pdf you provided but I am almost sure no one has been here for a while to change them.

Re: SAS VA Performing very slowly

pedrammobedi — Mon, 29 Aug 2016 13:26:03 GMT

I see that Java Platform SE Binary is in average consuming 30-40% of my CPU. It's an active process even when nobody is using SAS VA. Could this be a reason?

Re: SAS VA Performing very slowly

LinusH — Mon, 29 Aug 2016 13:33:38 GMT

I have no specific experience with VA/LASR logs, but 13 GB sounds quite large.

You may need to review your logging settings.

Also check the logs. There can be a lot of new types of warnings/errors that can be relevant in your investigation. I should be quite easy to see if the logging has elevated at the same time as your performance problems started.

Re: SAS VA Performing very slowly

pedrammobedi — Mon, 29 Aug 2016 13:41:31 GMT

There are many log files. Whch ones in specific should I check?

Also please take a look at the reply I posted about Java(TM) Platform SE Binary. Thanks!

Re: SAS VA Performing very slowly

JuanS_OCS — Mon, 29 Aug 2016 14:56:35 GMT

Hi,

1- as @LinusH said well, those big files on logs, even when no user is there and you already cleaned the logs, it seems to indicate that some of your log configurations are on too high detail level: http://support.sas.com/documentation/cdl/en/bimtag/68217/HTML/default/viewer.htm#p0qy7oulxf2ot7n1bcavt6difgu3.htm

You might want to change some levels from debug to error or info, what it was default.

2- can you get the full command line of the executable Java Platform SE Binary that consumes those resources?

a) this can help ups to locate and identify the process that might be an issue.

3- I believe to understand your VA installation is on Windows, in that case is non-distributed. How is your deployment looking like? All installed in 1 server? Metadata/Compute/Middle-tier separated?

4- Do you use the AutoLoad functionality or only Queries/Schemas?

5- Is it enabled recently the Performance Reports/ Environment Manager reports/APM/etc?

Re: SAS VA Performing very slowly

userid007 — Tue, 30 Aug 2016 00:24:33 GMT

as @JuanS_OCS mentioned, the critical issue would be the autoload logs under VA<config>Levn\Applications\SASVisualAnalytics\VisualAnalyticsAdministrator

I had to setup a clean up job to remove all the small log files as the number of files were huge.

perhaps cleanup your logs directory and if you are in VM's check if there is any cpu wait for your VM.

Re: SAS VA Performing very slowly

JuanS_OCS — Thu, 01 Sep 2016 09:30:31 GMT

Hi @pedrammobedi, since some days passed from your original question, I feel curious if you could make progress with your performance issues on VA. Any feedback will be welcome!

Best regards,

Juan

Re: SAS VA Performing very slowly

pedrammobedi — Thu, 01 Sep 2016 09:34:11 GMT

I have tried this to no avail.

Re: SAS VA Performing very slowly

pedrammobedi — Thu, 01 Sep 2016 09:39:31 GMT

Hi! Thank you for your response. Unfortunately the problem lingers. And I got more help from you people in the community that I received from SAS. They have simply directed me to learning Environment Manager in which I still do not see anything suspicious. All I know is that Java Platform SE binary is taking a lot of CPU even when nobody is using SAS. However, it is still hard to say what is causing it to work that hard. The only thing I see is my SAS VA taking a very long time to load. So any help is still appreciated.

Re: SAS VA Performing very slowly

JuanS_OCS — Thu, 01 Sep 2016 09:48:21 GMT

Hi,

sorry to learn that you still have issues there, but let's see if we can do something about it.

On one hand, I would keep open a track with SAS Technical Support and let them know that Environment Manager is not helping this time. It is a good tool, but not in your case for now.

In the other side, since most processes in SAS (metadata, compute and web tiers) are java processes, first we would need more information about which specific SAS process is giving you the high level of CPU usage, in order to filter from the huge amount of options available where the performance problem may came from.

For that, if you server is Lunux, ps -ef | grep java would help.

If your server is Windows, you can go to the Task Manager, and on the details tab, you can add the Command Line column and try to extend it as much as possible to get the full command.

Something else that may help is to know:

- Do you use AutoLoad or Data Builder Queries ?

- How your deployment looks like? How many servers compose the SAS VA application?

Re: SAS VA Performing very slowly

pedrammobedi — Thu, 01 Sep 2016 11:46:58 GMT

Hi. Thanks for you answer.

This is the full command for the Java prcess that takes most of CPU (the line highlighted in blue):

Let me know if you can read it. (righ click and open in new tab)

And your other questions:

Yes we use AUTOLOAD but there are only a few data sources and they have been there for a long time.
There is only one server for SAS VA afaik.

I also got a much better reply from SAS Support so I paste it below:

The JAVA memory use should on the other hand be related to WEB server activities

(I will check with some technical staff on this)

By the way could you check the following folder:

C:\SAS\Config\Lev1\Web\activemq\data - it should only contain 4 objects and the kahadb-folder content should be of current date and reasonable in size.

When I look at the kahaddb folder there is a log file called db-729.log that is about 32 MB. This file is used by Java and I need to stop the services in order to make any changes. Is this an unusual size for the file? Should I try getting a back up and replacing it with an empty one?

Re: SAS VA Performing very slowly

JuanS_OCS — Thu, 01 Sep 2016 11:54:08 GMT

Hi,

now it is clear to me that the problem is not on autoload, only the web application server, actially only on the SASServer1_1. And I know this problem, happens specially on the SAS Content Server and SAS Logon.

We are getting there.

Some more questions:

- Do you have only SASServer1_1 or also SASServer2_1 and SASServer12_1?

- Do you have enabled the Audit and Performance reports on VA? (So you have an AUDIT_ARTEFACTS table being created and loading into memory)? And you do, how big is this table at this moment?

- could you please share the JVM settings you have on the wrapper.conf on the SASServer1_1? (if you have more than 1_1, I would like to give a look into those).

Thank you in advance!

Looking forward for your feedback

Re: SAS VA Performing very slowly

pedrammobedi — Thu, 01 Sep 2016 12:08:21 GMT

Hi again,

There is only one web app in my services and that is SASServer1_1 - WebAppServer,
I doubt if there are any audits. At least I do not seem to find the audit_visualanalytics.sas7bdat file in my entire SAS folder.
This is the content of my wrapper.conf located in the sashome folder.

# Java Application
wrapper.java.command=E:\sas\sashome\SASPrivateJavaRuntimeEnvironment\9.4\jre\bin\java.exe

# Java Classpath (include wrapper.jar)  Add class path elements as
# needed starting from 1
wrapper.java.classpath.1=E:\sas\sashome\SASVersionedJarRepository\eclipse\plugins\JavaServiceWrapper_3.2.3\wrapper.jar
wrapper.java.classpath.2=E:\sas\sashome\SASVersionedJarRepository\eclipse\plugins\sas.launcher.jar

# Java Additional Parameters
wrapper.java.additional.1=-Djava.system.class.loader=com.sas.app.AppClassLoader
wrapper.java.additional.2=-Dsas.app.repository.path="E:\sas\sashome\SASVersionedJarRepository\eclipse"

Re: SAS VA Performing very slowly

userid007 — Thu, 01 Sep 2016 12:22:25 GMT

As @JuanS_OCS mentioned you might to focus on the Java heaps etc.

The audit analytics table will be under emdbla folder if you have enabled the load.

You might want to check the total time spent in garbage collection etx. Easiest way is to enable the auto load for va feed on emi frame work and you have default dashboards provided by sas one is for the application monitoring and another is for va usage.

Under application monitoring you might want to focus on app response time.

Best regards,
Sai korrapati

Re: SAS VA Performing very slowly

pedrammobedi — Thu, 01 Sep 2016 12:29:48 GMT

I am sorry, but could you be a little more specific like @JuanS_OCS? I haven't enabled AUDIT. And my SAS VA is currently too slow for me to start building reports for monitoring.

Kind regards,

Pedram