Hello,
after successful deployment of Campaign Management and Digital Marketing on new environment we have encountered performance issues with the aplication.
Application is often very slow and tasks like update, preview table, execute in simple campaign sometimes take insanely long - like 5mins, or 15 mins....
This behaviour fluctuate throughout the day, cannot say that the application is faster in early portion of the day (after everyday restart) and slower in the afternoon. It's just overall slow with some deviations.
These are the steps we took to address the issue:
- Installed the latest hotfixes:
DM6.5
http://ftp.sas.com/techsup/download/hotfix/HF2/Z69.html
MA6.5
http://ftp.sas.com/techsup/download/hotfix/HF2/Z77.html
- Worked on the stability of Object Spawner increasing the number of processes for campaigns (multibridge connections on STP server).
- Increased heap memory to recommended values and often higher values. (http://support.sas.com/documentation/cdl/en/appsrvtuning/69859/HTML/default/viewer.htm#n1ov80pk0vgic...)
- Increased the MEMSIZE to 8G on Compute server.
After having a word with colleague who works with the application, i get an impression that maybe there is some problem with DB communication, because in my opinion application isn't really continually using up CPU that much.
SASCustIntelCore6.5.log sometimes generate this type of error:
2019-06-16 09:45:22,504 ERROR [CIExec-15] [d065c7aace414180:774ba052:16b5d9cd45a:8d9] [E546488] ata.datapattern.MATableUploadManagerImpl - TID[CIExec-15] Error trying to upload tables:
com.sas.analytics.crm.error.client.ApplicationException: Problem uploading.
at com.sas.analytics.crm.custdata.datapattern.MATableUploadManagerImpl.callTableMGMTMacro(MATableUploadManagerImpl.java:394)
at com.sas.analytics.crm.custdata.datapattern.MATableUploadManagerImpl.uploadTables(MATableUploadManagerImpl.java:185)
at com.sas.analytics.crm.custdata.datapattern.MAIQService.uploadProcess(MAIQService.java:3113)
at com.sas.analytics.crm.custdata.datapattern.MAIQService.uploadHT(MAIQService.java:3135)
at com.sas.analytics.crm.custdata.datapattern.MAIQService.executeUploadHT(MAIQService.java:3379)
at com.sas.analytics.crm.custdata.datapattern.MAIQService.execute(MAIQService.java:3508)
at com.sas.analytics.crm.custdata.datapattern.MAQuery.doExecute(MAQuery.java:781)
at com.sas.analytics.crm.task.ejb.Task.execute(Task.java:167)
at com.sas.analytics.crm.task.ejb.Task.executeFromTaskManagement(Task.java:127)
at com.sas.analytics.crm.task.ejb.Task.run(Task.java:469)
at com.sas.analytics.crm.task.ejb.TaskSequence$WrappedTask.run(TaskSequence.java:1398)
at com.sas.analytics.crm.task.queue.JobRunner.run(JobRunner.java:32)
at com.sas.analytics.crm.task.ejb.ThreadPoolPriorityExecutorImpl$QueueExecuteHandler.run(ThreadPoolPriorityExecutorImpl.java:113)
at com.sas.analytics.crm.task.ejb.ClearedThreadLocalRunnable.run(ClearedThreadLocalRunnable.java:17)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1152)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:622)
at java.lang.Thread.run(Thread.java:748)
2019-06-18 11:14:10,828 WARN [tomcat-http--48] [01edd0f21a6fd1e9:-2d90f95c:16b67e98b4e:-b1d] [E542313] support.RemoteInvocationTraceInterceptor - Processing of HttpInvokerServiceExporter remote call resulted in fatal exception: com.sas.analytics.crm.cm.client.CampaignProxy.getCampaignDetails
com.sas.ci.services.common.error.CIException: Your session is no longer available.
at com.sas.analytics.crm.ejb.BaseProxyImpl.handleException(BaseProxyImpl.java:606)
at com.sas.analytics.crm.cm.ejb.CampaignProxyImpl.runFIFO(CampaignProxyImpl.java:4270)
at com.sas.analytics.crm.cm.ejb.CampaignProxyImpl.runFIFO(CampaignProxyImpl.java:4254)
at com.sas.analytics.crm.cm.ejb.CampaignProxyImpl.getCampaignDetails(CampaignProxyImpl.java:326)
at sun.reflect.GeneratedMethodAccessor2179.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at org.springframework.remoting.support.RemoteInvocationTraceInterceptor.invoke(RemoteInvocationTraceInterceptor.java:77)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at com.sun.proxy.$Proxy552.getCampaignDetails(Unknown Source)
at sun.reflect.GeneratedMethodAccessor2178.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.springframework.remoting.support.RemoteInvocation.invoke(RemoteInvocation.java:205)
at org.springframework.remoting.support.DefaultRemoteInvocationExecutor.invoke(DefaultRemoteInvocationExecutor.java:38)
at sun.reflect.GeneratedMethodAccessor186.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)........
Can i ask for some tips or measures to further increase the performance?
Application runs on 4 nodes - metadata, compute, midtier, and VA server. Connections between servers should be without problem.
Clients using the application are few - there is maximum of 5-7 clients at a time connected.
I will be thankful for any advices.
Best regards,
Lukas.
