SAS MA performance is very slow

[fatcat]

Hi, users from time-to-time experience slow perfomance saving and opening a campaign within MA. There are ample of space (grid), CPU, and the datasource traffic looks to be very light. 

Any idea?

Do notice the LDAP server is low on space. Could this impact the overall MA performance?

TIA, 

[pcapazzi]

CIS is run inside of a JVM. Are you looking at the statistics of the JVM? The Server's O/S may show everything as ok but the JVM is it's own container with a dedicated amount of resources.

We had an issue with performance which we attributed to low heap space in the JVM. In the CIS Admin Guide (pg 219) there is be a reference to the Java parameter -Xmx. After a couple of times of increasing that value we have not experience frequent slowdowns. If I recall right we doubled the value 2 times. (ex: original -Xmx2gb 2nd try -Xmx4gb -Xmx 8gb). 

Hope that helps. 

[shill]

I agree that 2GB is probably not enough, but do be a little cautious with increasing the max heap size too much, as you might experience some intermittent performance hits when garbage collection occurs. We try to keep it to no more then 8GB, and only go that high if 6GB isn't sufficient. You might also want to increase -Xms at the same time so you don't have performance issues when new memory needs to be allocated.

The other thing that can cause seemingly simple things to intermittently take a long time is if you don't have enough load balanced connections defined on the Stored Process Server that's supporting MA (if you have just one, it's SASApp, but depending on your installation, you might have more than one and could have a "SASCI" or "SASAMA", etc. server context). The out-of-the-box configuration includes 3 that are configured to handle multiple concurrent jobs, but we like to re-configure that to have ~20-30 that handle just one job at a time, which avoids short jobs getting queued up behind longer-running ones. Check out http://support.sas.com/kb/40/567.html for more details, and contact support@sas.com if you have questions about this.

[JuanS_OCS]

Hi @fatcat,

I agree with @pcapazzi and @shill, I think they are good proposals:

- increase the memory and max number of connections for the  JVM on SASServer1 (your SCS) and the SASServer of MA/MO and restart the middle tier.

- increase the number of load balanced STPs and restart the Object Spawner.

Additionally, I have seen, specially on MA/MO solutions, and because UTF-8 is working on the SASApp, you may have additional activity on your Database, just querying but really doing nothing. You might want to review the activity on your database. This should not happen, but I actually saw this surprisingly happening after a migration from CI 5 to 6, and using a different database engine.

As said, not necessarily your case, but who knows.

If you find this to happens, I strongly recommend you to get aligned with your best DBA expert and SAS Technical Support.

[fatcat]

Hi All, thanks for all the replies. Would the issue show up in any of the log files?  The Production team would need to have definitive 'evidence' that is the root cause in order to make the change in Production. 

Our DBA did pull stats on the DB side, looks pretty clean.

[pcapazzi]

If it's not already implemented you could use the SAS Environment Manager
to collect data about the JVM. I have NOT worked with it myself but I have
seen a chart from it showing how memory was trending downwards in our case.

If you do not have this implemented perhaps jmonitor could help give some
insight.

##- Please type your reply above this line. Simple formatting, no
attachments. -##

[fatcat]

@pcapazzi - we have it installed but not configured (SAS PS is coming to do that), so when there are error pages everywhere within the app. 

Did you have an attachment, doesn't look like it came thru. 

[pcapazzi]

I did not have an attachment of the chart. 

[Birendra]

Hi,

There could be various reason of MA slowness specially while opening and saving campaigns.

1. I/O bottleneck with SAS content server.

2. The folder where the campaign being save having too much object(campaigns, export definitions etc saved in same folder)

3. Network slowness between client to SAS servers.

4. JVMs Memory size (Min, Max and Heap size)

Suggestion:

1. could you please run the I/O test on your Mid and compute server and check how I/O result is there.

2. Create a new folder and try to save the campaign in that new folder and see how it goes?

3. if above 2 are not helped, then go to increase the JVMs memory

Thanks,

Birendra