CLOSE_WAITS increase causing WEBAPP server latency

[sandeep_reddy]

HI Team,

I am facing issues with CLOSE_WAITS in Web Server. Whenever, i see this number incraeses to > 50  WebAppServer latent/stops responding to the requests.

We have JVM heap memory setting to 16GB and garbage collection is enabled. Initially we suspect that GC is causing this issue but not enough evidences too prove the same.

My Webserver version is 2.2. Please let me know , what are the reasons of increasing this close_waits number.

Best Regards,
Sandeep

[Kurt_Bremser]

If this is happening in the SAS-supplied httpd, then open a track with SAS technical support.

Otherwise, do a google search for "httpd close_wait" and see if you find something relating to your situation. Most often it's a problem in the webapp.

[nhvdwalt]

Hi @sandeep_reddy 

Is this on Windows ?

[sandeep_reddy]

Hello,

This is on Unix and SAS Version 9.4 M3.

Warm Regards,

Sandeep

[nhvdwalt]

Ok, just wanted to check since I've seen this before on Windows.

Agree with @Kurt_Bremser , best is to open a Track with Tech Support.

[sandeep_reddy]

Hello,

I have opened a SAS track and below is the update.

1) After investigating on tcp dump and webserver worker threads SAS support asked us to upgrade the WebServer 2.2 to 2.4

2) We have done the same and observing the same behavior.

Any small suggestions in fine tuning as well, happy to test in my environments. Need a direction to close this issue.

Best Regards,
Sandeep

[AndreasWindisch]

Hello,

have you noticed that there are some suggestions for Operation System Tuning in the

 
SAS® 9.4 Web Applications: Tuning for Performance and Scalability
http://support.sas.com/documentation/cdl/en/appsrvtuning/69859/HTML/default/viewer.htm#n0xfno04p3q3u... ?

If you are on Red Hat you can have a look to 'Red Hat Enterprise Linux Network Performance Tuning Guide', there a serveral parameters which can infulence the behaivour on the network interfaces.

Best regards,
Andres

[sandeep_reddy]

Hello,

Thanks a lot for your response.

All the limits in /etc/sysctl.conf has been modified to highest number to handle the load.

net.ipv4.tcp_max_syn_backlog: 4096

net.core.somaxconn:6000

net.core.netdev_max_backlog:3000

net.ipv4.tcp_keepalive_time=300
net.ipv4.tcp_keepalive_intvl=60
net.ipv4.tcp_keepalive_probes=20
net.ipv4.tcp_fin_timeout=30
net.core.netdev_max_backlog=3000

What surprises me is that, WebServer has already send ACK to Load balancer for close request. I could see in ssl webserver logs as POST /SASLogon/ClearPass entries.

Even my TCP dump output shows the same. But still my Webserver is not able to reuse/clear the unused sockets and it seems it is still waiting for something(Unknown).

We have 300 users and 6 JVMS with 16GB heap memory. I am not sure where this bottleneck exists.

Thanks & Regards,
Sandeep