CPU spikes as soon as DataFlux Data Management Server 2.6 starts queuing real time service requests.

[Che]

Hello,

We have SAS DataFlux Data Management server-2.6 hosted on Linux OS. It is configured to run 30 real time DFWSVC processes and have 4 CPUs/50GB RAM.

Recently we have noticed our real time service load has increased on this server and when all 30 real time DFWSVC processes are busy processing real time service requests, all new requests are queued by server. As soon as queuing is triggered, we see CPU utilization jumps to 95-99%. Due to this DataFlux application becomes unhealthy and responds very slowly to incoming requests.

Our average CPU utilization with out any queuing is 10-30%.

I am curious if any other SAS users have seen this behavior and what action they have taken to resolve this. For SAS, is this a bug or something for which we need a patch?

Any help will be greatly appreciated.

Chetan

3095316661

[RonAgresta]

Hi Chetan - 

This behavior may indicate that the resources available to the server processing the services is not adequately sized to handle the load when all 30 services are engaged. You could try turning off queuing on DM Server, but then the calling applications would have to deal with real-time services that were denied a slot for processing on DM Server (the calling app would have to retry failed service calls).

Take a look at your memory utilization when all 30 real-time services are engaged, this is usually a good indicator of whether DM Server is OK with resources.

Ron

[Che]

Hello Ron,

Thanks for your reply.

I believe memory was at it's peak when that event took place. Our CPU and memory utilization all the time is well below 30%(meaning resources are under utilized), it spikes up only when queuing is triggered.

So talking about queuing, do you know if there is limit on how many service requests DM server can queue at any given point of time and where does application stores that data when service requests starts to queue?

Best regards,

Chetan

[RonAgresta]

Take a look at the DM Server documentation. There are some settings regarding queues and one in particular, DMSERVER/SOAP/ CONNS_BACKLOG, that may help you manage an overloaded server. Also ensure you have all DM Server patches installed.

There are also ways to streamline the footprint of your services. Many nodes have memory settings for example and optimizing these settings may help you shrink services so that they are not so resource intensive.

Ron

[Che]

Hello Ron,

Thanks again for your help and response with my questions.

DMSERVER/SOAP/CONNS_BACKLOG

Specifies the maximum size of the connection request queue. The queue size limit enables the SOAP server to refuse connection requests when it can no longer process them within an acceptable period of time. The default is 100.

I am not very clear with this info and need more info. Is this putting a limit on queue size or it is putting limit on connection requests?

I ask as as we do not have this parameter set in our dmserver.cfg file and we saw more than 1000 requests queued at one point.

Another update I would like to give you is we increased max no.of DFWSVC processes that can run simultaneously from current 30 to 60 and yesterday we noticed server used 35 processes at one point and we did not run in to any issue from CPU and Memory usage  perspective. Both were under 30% and server was in healthy state. So this let me believe that CPU/Memory jumps up only when queuing  is triggered.

I checked all available hot fixes for DM server 2.6 and did not find one which has any correlation to queuing issue.

So please advice how should we deal with and engage SAS  as it is causing client facing impact.

Thanks again.

Regards,

Chetan

[RonAgresta]

Please contact SAS technical support to ensure you are getting the help you need.

Ron