BookmarkSubscribeRSS Feed
Che
Fluorite | Level 6 Che
Fluorite | Level 6

Hello,

 

We have SAS DataFlux Data Management server-2.6 hosted on Linux OS. It is configured to run 30 real time DFWSVC processes and have 4 CPUs/50GB RAM.

 

Recently we have noticed our real time service load has increased on this server and when all 30 real time DFWSVC processes are busy processing real time service requests, all new requests are queued by server. As soon as queuing is triggered, we see CPU utilization jumps to 95-99%. Due to this DataFlux application becomes unhealthy and responds very slowly to incoming requests.

Our average CPU utilization with out any queuing is 10-30%.

 

I am curious if any other SAS users have seen this behavior and what action they have taken to resolve this. For SAS, is this a bug or something for which we need a patch?

 

Any help will be greatly appreciated.

 

Chetan

3095316661

 

 

5 REPLIES 5
RonAgresta
SAS Employee

Hi Chetan - 

 

This behavior may indicate that the resources available to the server processing the services is not adequately sized to handle the load when all 30 services are engaged. You could try turning off queuing on DM Server, but then the calling applications would have to deal with real-time services that were denied a slot for processing on DM Server (the calling app would have to retry failed service calls).

 

Take a look at your memory utilization when all 30 real-time services are engaged, this is usually a good indicator of whether DM Server is OK with resources.

 

Ron

Che
Fluorite | Level 6 Che
Fluorite | Level 6

Hello Ron,

 

Thanks for your reply.

 

I believe memory was at it's peak when that event took place. Our CPU and memory utilization all the time is well below 30%(meaning resources are under utilized), it spikes up only when queuing is triggered.

 

So talking about queuing, do you know if there is limit on how many service requests DM server can queue at any given point of time and where does application stores that data when service requests starts to queue?

 

Best regards,

Chetan

RonAgresta
SAS Employee

Take a look at the DM Server documentation. There are some settings regarding queues and one in particular, DMSERVER/SOAP/ CONNS_BACKLOG, that may help you manage an overloaded server. Also ensure you have all DM Server patches installed.

 

There are also ways to streamline the footprint of your services. Many nodes have memory settings for example and optimizing these settings may help you shrink services so that they are not so resource intensive.


Ron

Che
Fluorite | Level 6 Che
Fluorite | Level 6

Hello Ron,

 

Thanks again for your help and response with my questions.

 

DMSERVER/SOAP/CONNS_BACKLOG
Specifies the maximum size of the connection request queue. The queue size limit enables the SOAP server to refuse connection requests when it can no longer process them within an acceptable period of time. The default is 100.

 

I am not very clear with this info and need more info. Is this putting a limit on queue size or it is putting limit on connection requests?

I ask as as we do not have this parameter set in our dmserver.cfg file and we saw more than 1000 requests queued at one point.

 

Another update I would like to give you is we increased max no.of DFWSVC processes that can run simultaneously from current 30 to 60 and yesterday we noticed server used 35 processes at one point and we did not run in to any issue from CPU and Memory usage  perspective. Both were under 30% and server was in healthy state. So this let me believe that CPU/Memory jumps up only when queuing  is triggered.

 

I checked all available hot fixes for DM server 2.6 and did not find one which has any correlation to queuing issue.

 

So please advice how should we deal with and engage SAS  as it is causing client facing impact.

 

Thanks again.

 

Regards,

Chetan

RonAgresta
SAS Employee

Please contact SAS technical support to ensure you are getting the help you need.

 

Ron

sas-innovate-2024.png

Don't miss out on SAS Innovate - Register now for the FREE Livestream!

Can't make it to Vegas? No problem! Watch our general sessions LIVE or on-demand starting April 17th. Hear from SAS execs, best-selling author Adam Grant, Hot Ones host Sean Evans, top tech journalist Kara Swisher, AI expert Cassie Kozyrkov, and the mind-blowing dance crew iLuminate! Plus, get access to over 20 breakout sessions.

 

Register now!

How to connect to databases in SAS Viya

Need to connect to databases in SAS Viya? SAS’ David Ghan shows you two methods – via SAS/ACCESS LIBNAME and SAS Data Connector SASLIBS – in this video.

Find more tutorials on the SAS Users YouTube channel.

Discussion stats
  • 5 replies
  • 863 views
  • 0 likes
  • 2 in conversation