Performance optimization loading VA reports in Modern Viewer

[AreSivertsen]

Hi

I'm trying to decrease the time a VA reports takes to load in the web browser.

System is:

• SAS 9.4 M5
• Visual Analytics 7.4
• Mid-Tier is a single Windows Server 2012R2
• Modern Viewer 

I have chosen a medium sized report for testing. I have tested several browsers and found that Chrome is performing clearly best with an average of 11 seconds from clicking the link to the report is fully loaded.(Edge 45s, Firefox 21s).

The Loading screen

shows for 5 seconds and the "dotted" screen shows the remaining 6 seconds

If I use the Developer tool in Chrome(F12) I can see that the loading stops at two points. The first time for 4 seconds

and the second time for 1 second.

Both are 404(Not Found) error messages and both are related to the default SAS Corporate theme.

I'm not very familiar to web debugging and really need some expert help here. Anyone who can give some tips on how I should proceed?

Best regards

Are Sivertsen

[utrocketeng]

following....

[JuanS_OCS · Accepted solution]

Hello @AreSivertsen, @utrocketeng,

this is a great question. In SAS VA, you need to be mindful that slow performance might come from different areas: report, application JVM, overloading due to WIP database (normally produced if you have too big audit tables in the database), etc.

For the reports themselves, you would see the problem when you can actually see how the report loads, and loads the data, but very slowly. For that purpose, you have a performance debugger in SAS VA Report Designer. See:

- http://support.sas.com/resources/papers/proceedings17/SAS0734-2017.pdf

- https://support.sas.com/resources/papers/proceedings15/3275-2015.pdf

Additionally, the web browser developer tools (Chrome, Firefox, IE), are very useful. For those 404s is better to contact SAS Technical Support. You can also see the "Network" tab, where you can see the time that each component takes.

For the application JVMs, the best is to monitor the status of the different JVMs, with JMX monitors. SAS Environment Manager offers you this option, but you can also do by yourself. And I recommend to do some fine tuning on the middle tier JVMs. Please see:

- https://communities.sas.com/t5/SAS-Visual-Analytics/SAS-VA-Performing-very-slowly/td-p/294713

For the WIP and audit tables, please check first if you actually enabled this functionality. If you did, you might want to see how to archive this data and do some cleaning from time to time. If this happens, basically you will see how the JVM of SASServer1_1 fills up and conclussion is that, either you extend the Jmx values or you clean audit tables:

- http://support.sas.com/kb/58/599.html

- https://blogs.sas.com/content/sgf/2016/01/26/going-visual-analytics-audit-data-archiving/

- https://blogs.sas.com/content/sgf/2015/11/09/controlling-the-size-of-the-web-infrastructure-platform...

- http://support.sas.com/kb/60/182.html

Edit: if you have SAS 9.4 M5, you probably have SAS VA 7.5, not 7.4. And if you have VA 7.4, you would have probably SAS 9.4 M4. 

[AreSivertsen]

Thank you for the information so far. This gives me something to start working on.I will start in our DEV environment tomorrow.

Regarding the report improvements I will have to wait until our report designer comes off vacation in August. I have done some testing this week and have some suggestions to improve performance.

JVM I will start testing tomorrow, but first: I checked the wrapper.conf file and these are the settings:

wrapper.java.additional.7=-Xmx4096m
wrapper.java.additional.8=-Xss256k
wrapper.java.additional.9=-Dsas.deployment.agent.client.config="E:/SAS/SASHome/SASRemoteDeploymentAgentClient/2.1/config/deployagtclt.properties"
wrapper.java.additional.10=-Dsas.auto.publish.host=sas-md-prod.fisk.intern
wrapper.java.additional.11=-XX:MaxPermSize=1024m

.9 and .10 refers to files/servers instead of values, as it does in your post. I think its "out of the box" when installing. Any reasons why I shouldn't change them?

When it comes to Audit tables, they are enabled. But extracting and purging is set up, I think. SAS Norway did this for us, but I will check anyway

Thanks again

Are

[JuanS_OCS]

You are very welcome. Hope it can help you on any aspect.

About the JVM, the line numbers does not matter much, after  new versions, hotfixes, manual changes ... the line numbers may change. What is important is the variable defined after "wrapper.java.additional.N=". E.G: -Xms or -Xmx values. The tuning guidelines you can find them here https://support.sas.com/documentation/cdl/en/appsrvtuning/69859/HTML/default/viewer.htm#p1tpeih89ioi...

Indeed, files/server is not much relevant, but the memory limits of the JVM, heap sizes, are.

[AreSivertsen]

Hi Juan

I have checked my wrapper.conf in our PROD-evironment. My settings are:

SAS_Server1_1 Wrapper.conf

wrapper.java.additional.54=-Xms1024m
wrapper.java.additional.7=-Xmx4096m
wrapper.java.additional.38=-XX:PermSize=512m
wrapper.java.additional.11=-XX:MaxPermSize=1024m

SAS_Server 12_1 Wrapper.conf

wrapper.java.additional.44=-Xms8192m
wrapper.java.additional.7=-Xmx8192m
wrapper.java.additional.32=-XX:PermSize=1280m
wrapper.java.additional.10=-XX:MaxPermSize=1280m

The settings in the SAS_Server1_1 file is maybe to low. But none of them have settings for

• Dsas.svcs.http.max.total.connections
• Dsas.svcs.http.max.connections
• maxPoolSize
• maxThreads

Are these necessary?

Why is it the server1_1 file that is used? If I should have guessed I would  have guessed the server12_1 file(Visual Analytics)

Best regards

Are

[JuanS_OCS]

Hello @AreSivertsen,

I would definetely increase the values, if your RAM memory has still that free space available.

And yes, I would also increase the pool of connections, as best practice (again, if RAM allows). 

The last settings are for the server.xml not the wrapper.conf, and you could check also the settings for your Apache/Web Server.

Actually, that pool, sometimes needs to be even larger than just 300 or 500. It all depends on what your environment and its actual usage is requesting to the system. And this is why the monitoring of the JVMs is that important.

[SASKiwi]

We just went through a very similar performance tuning exercise ourselves, and your performance stats with the Modern Viewer mirror ours - the Modern Viewer is slow to load reports but fast to navigate them. We find the Classic Viewer fast to load and fast to navigate.

How does your Classic Viewer performance compare? If your Classic Viewer report load and navigation times are a lot faster and very acceptable I think it is safe to assume that your VA servers don't need much in the way of tuning and this is more of a browser problem. It doesn't do any harm to check out how your JVMs are running though. We found that ensuring all JVMs have a lot of free memory was one of the keys to good server performance. Check out your VA Environment Manager performance stats as a starting point. Note any regular alerts can be an indication of non-optimal performance.  

[AreSivertsen]

Hi SASkiwi

You brought my attention to testing the Classic Viewer. It actually performs slower than Modern Viewer. So I think I will make little more effort to optimizing the server

Here are my results:

Classic Viewer:

• Chrome 16,5s
• Edge 16s

Modern Viewer:

• Chrome 11,5s
• Edge 40,3s
• Firefox Quantum 22,7s
• Opera 11,9s
• Safari on Mac 10,0s

I'm a bit surprised that Safari performed so well.

The testing was done by measuring 31 reports and taking the average. I attach a spreadsheet with the testdata in case anyone is interested. Report names are in Norwegian, though :- )

Best regards

Are

[SASKiwi]

That's very interesting. I agree that checking out server performance would indeed be worthwhile in your case.

BTW, I would recommend engaging SAS Tech Support also if you haven't done so already. They are the experts in analysing server and browser logs to identify bottlenecks / performance issues. They were very helpful in fixing our performance issues.

[SASKiwi]

Here is a setting change to Server1_1 that improved performance a lot for us. Here is a link describing this:

http://support.sas.com/documentation/cdl/en/appsrvtuning/69859/HTML/default/viewer.htm#n11d31ckagwav...

The setting from this page is, assuming you are still using the default value of 500 (note another symptom of poor server performance is your tmlog*.log files are very large).

Modify the <SAS_CONFIG>\Lev1\Web\WebAppServer\SASServer1_1\lib\jta.properties file and add/modify the following line at the bottom of that file:
com.atomikos.icatch.checkpoint_interval=50

Of course you will have to stop and start the server for the setting to take effect, and it is worthwhile clearing out the LOG directory at the same time.

[JuanS_OCS]

Some additions from what I remember and I think it might help:

HTML5 was introduced in SAS VA 7.3 as pre-production and in 7.4 as production. Still, in 7.5 there were a lot of improvements, but also a few bugs that were solved by hotfixes and understanding how the web browsers do work (and the differences between them).

Problem Note 56894: Large SAS® Visual Analytics reports might perform poorly http://support.sas.com/kb/56/894.html

Problem Note 60665: When running in Microsoft Internet Explorer 11, the modern version of SAS® Visual Analytics Viewer might initially take a long time to open a report http://support.sas.com/kb/60/665.html

Problem Note 57733: Usage of SAS® Visual Analytics in modern mode can cause the web browser to stop responding when it is low on memory http://support.sas.com/kb/57/733.html

Problem Note 56894: Large SAS® Visual Analytics reports might perform poorly 

For browsers:

https://stackoverflow.com/questions/985431/max-parallel-http-connections-in-a-browser

https://en.wikipedia.org/wiki/Comparison_of_web_browsers

https://www.webperformance.com/load-testing-tools/blog/2018/02/what-is-the-fastest-web-browser-in-20...

Bottom line is:

Besides the previous comments, please consider also and compare web browsers, ensuring they are 64-bit based (IE, won't give it, since background process are always 32-bit) and ensure the latest hotfixes are available to SAS (and browser). Maintenance 5/SASVA 7.5 will help as well with the latest improvements.

Kind regards,

Juan

[AreSivertsen]

HI

I'm finished(for now) tuning for performance. 

Thanks to Juan and SASKiwi for the contributions.

Reports are now running in average 3.4 seconds faster in Chrome, which is the fastest of the most common browsers.

The largest improvements came from rescheduling jobs and stopping unnecessary services on the mid-tier server and from reconfiguring JVM.

SASKiwi: I haven't yet testing reducing the checkpoint_interval setting, but will do so when I'm back from a one week vacation

Thanks Again

Are

[SASKiwi]

@AreSivertsen - I'd be interested in knowing the JVM config changes. On our system we greatly increased the Server1_1 memory settings:

wrapper.java.additional.7=-Xmx12288m
wrapper.java.additional.9=-Xms8192m

This gave us a lot more free memory for this JVM server, which also reduced the CPU usage on our SAS mid-tier server.

[AreSivertsen]

Hi SASKiwi

I changed the settings from:
wrapper.java.additional.7=-Xmx4096m
wrapper.java.additional.54=-Xms1024m

wrapper.java.additional.11=-XX:MaxPermSize=1024m

wrapper.java.additional.38=-XX:PermSize=512m

to 

wrapper.java.additional.7=-Xmx8192m
wrapper.java.additional.54=-Xms8192m

wrapper.java.additional.11=-XX:MaxPermSize=1280m

wrapper.java.additional.38=-XX:PermSize=1280m

This improved performance by in average up to a second loadtime per report

At a later time I  added

com.atomikos.icatch.checkpoint_interval=50

to the jta.properties file. I couldn't measure any increase in performance with this change

Best regards
Are