topic Re: CLOSE_WAITS increase causing WEBAPP server latency in Administration and Deployment

CLOSE_WAITS increase causing WEBAPP server latency

sandeep_reddy — Mon, 14 Oct 2019 06:38:42 GMT

HI Team,

I am facing issues with CLOSE_WAITS in Web Server. Whenever, i see this number incraeses to > 50 WebAppServer latent/stops responding to the requests.

We have JVM heap memory setting to 16GB and garbage collection is enabled. Initially we suspect that GC is causing this issue but not enough evidences too prove the same.

My Webserver version is 2.2. Please let me know , what are the reasons of increasing this close_waits number.

Best Regards,
Sandeep

Re: CLOSE_WAITS increase causing WEBAPP server latency

Kurt_Bremser — Mon, 14 Oct 2019 07:08:55 GMT

If this is happening in the SAS-supplied httpd, then open a track with SAS technical support.

Otherwise, do a google search for "httpd close_wait" and see if you find something relating to your situation. Most often it's a problem in the webapp.

Re: CLOSE_WAITS increase causing WEBAPP server latency

nhvdwalt — Mon, 14 Oct 2019 07:46:27 GMT

Hi @sandeep_reddy

Is this on Windows ?

Re: CLOSE_WAITS increase causing WEBAPP server latency

sandeep_reddy — Mon, 14 Oct 2019 10:04:51 GMT

Hello,

This is on Unix and SAS Version 9.4 M3.

Warm Regards,

Sandeep

Re: CLOSE_WAITS increase causing WEBAPP server latency

nhvdwalt — Mon, 14 Oct 2019 11:30:39 GMT

Ok, just wanted to check since I've seen this before on Windows.

Agree with @Kurt_Bremser , best is to open a Track with Tech Support.

Re: CLOSE_WAITS increase causing WEBAPP server latency

sandeep_reddy — Fri, 25 Oct 2019 00:24:22 GMT

Hello,

I have opened a SAS track and below is the update.

1) After investigating on tcp dump and webserver worker threads SAS support asked us to upgrade the WebServer 2.2 to 2.4

2) We have done the same and observing the same behavior.

Any small suggestions in fine tuning as well, happy to test in my environments. Need a direction to close this issue.

Best Regards,
Sandeep

Re: CLOSE_WAITS increase causing WEBAPP server latency

AndreasWindisch — Mon, 28 Oct 2019 07:15:20 GMT

Hello,

have you noticed that there are some suggestions for Operation System Tuning in the

SAS® 9.4 Web Applications: Tuning for Performance and Scalability
http://support.sas.com/documentation/cdl/en/appsrvtuning/69859/HTML/default/viewer.htm#n0xfno04p3q3urn1513f8zdjx72m.htm ?

If you are on Red Hat you can have a look to 'Red Hat Enterprise Linux Network Performance Tuning Guide', there a serveral parameters which can infulence the behaivour on the network interfaces.

Best regards,
Andres

Re: CLOSE_WAITS increase causing WEBAPP server latency

sandeep_reddy — Wed, 30 Oct 2019 01:14:43 GMT

Hello,

Thanks a lot for your response.

All the limits in /etc/sysctl.conf has been modified to highest number to handle the load.

net.ipv4.tcp_max_syn_backlog: 4096

net.core.somaxconn:6000

net.core.netdev_max_backlog:3000

net.ipv4.tcp_keepalive_time=300
net.ipv4.tcp_keepalive_intvl=60
net.ipv4.tcp_keepalive_probes=20
net.ipv4.tcp_fin_timeout=30
net.core.netdev_max_backlog=3000

What surprises me is that, WebServer has already send ACK to Load balancer for close request. I could see in ssl webserver logs as POST /SASLogon/ClearPass entries.

Even my TCP dump output shows the same. But still my Webserver is not able to reuse/clear the unused sockets and it seems it is still waiting for something(Unknown).

We have 300 users and 6 JVMS with 16GB heap memory. I am not sure where this bottleneck exists.

Thanks & Regards,
Sandeep