SASServer1_1 takes more time than usual for starting up

[sandeeppajni1]

Dear Folks,

We are using SAS 9.4M6 with having multiple tier architecture (3 meta, 6 compute and 2 web nodes - all in Linux). We are facing issues in turning on the web services, specifically SASServer1_1 which takes time more than usual. While trying to turn on the services, they are shown like below. However, I read somewhere that in 2 hours everything is up, SAS Studio works absolutely fine after a couple of hours. Cleaning of logs did not help.

I am looking for a reason of such a delay, and possible solution to fix it. A call has already been raised with SAS tech support and I am currently awaiting a response from them.

/opt/sas/web/sasconfig/Lev1/sas.servers start

Starting SAS servers

Starting SAS JMS Broker

Starting SAS Cache Locator Service: ins_41415

waiting 30 ...

Starting SAS Web Server

Starting SAS Web App Server: SASServer1_1

(this can take a while)

waiting 30 ...

waiting 30 ...

waiting 30 ...

waiting 30 ...

waiting 60 ...

waiting 60 ...

waiting 60 ...

waiting 60 ...

waiting 60 ...

waiting 60 ...

waiting 60 ...

waiting 60 ...

waiting 60 ...

waiting 60 ...

waiting 60 ...

waiting 60 ...

waiting 60 ...

waiting 120 ...

waiting 120 ...

waiting 120 ...

waiting 120 ...

waiting 120 ...

waiting 120 ...

SAS Web App Server SASServer1_1 is still in the process of starting up,

proceeding with additional servers.

Starting SAS Web App Server: SASServer2_1

(this can take a while)

waiting 30 ...

waiting 30 ...

Starting SAS Environment Manager

Starting SAS Environment Manager Agent

While checking the the logs, it shows below such messages in the catalina file

[pool-7-thread-1] WARN com.sas.svcs.cluster.httpinvoker.AutoDiscoveredHttpService - Have received 20 failures while trying to access an auto-discoverable service to satisfy request for 'com.sas.svcs.config.client.ConfigurationServiceInterface'. Please ensure that the application and/or process surfacing this interface is still in the process of starting.

[SASKiwi]

Is there anything unusual in the SASServer1_1 log during the startup period? What was the SAS web server CPU performance like prior and during the startup? Typically these startups consume 100% of the CPU for a few minutes starting a whole bunch of Java processes. 

[gwootton]

Technical Support I think will be your best path here as they'll be able to take a look at your logs to identify if a specific application is causing the delay and help with tuning. You could try increasing the memory allocation for SASServer1 in the /opt/sas/web/sasconfig/Lev1/Web/WebAppServer/SASServer1_1/bin/setenv.sh file (the Xmx and Xms settings in the JVM_OPTS environment variable).

[sandeeppajni1]

Hi @gwootton,

Thank you for suggestions. I am already actively working with the SAS tech support to know the bottom of this issue. Setenv.sh has been already verified, it as more than enough space (4 GB each for xms and xmx parameters)

[sandeeppajni1]

Hi All,
This is just to update further on this issue, it was identified that SASDeploymentBackup.xml service while turning on the SASServer1_1 is taking most of the time. In logs, it was identified that deployment backup tries to connect to the client tiers which are not reachable somehow. Recently, we installed SAS EG on new client machines which are creating problems as those made entry in sas metadata. We checked that deployment agents are running fine in the client nodes but then why client tiers are not reachable for sas deployment backup for SASServer1_1. Anyone has any idea about it?
Also, does sas count on client machines and their deployment agents as well?

[gwootton]

If the delay is coming from the SAS Deployment Backup application you may wish to examine the logs for that specific application to see if it has additional detail on the source of the delay (Lev1/Web/Logs/SASServer1_1). I would also mention that 4 GB is not necessarily enough, I've had to increase this from 4 GB to 8 GB or 16 GB in some of my environments.

[sandeeppajni1]

Hi @gwootton,

The delay is happening in SAS Deployment Backup as it tries to reach out the newly added client tiers (SAS Deployment Agents services are running in client tiers too) whenever reboot of middle tier services happen. Somehow, it shows that the client tiers (Windows Virtual Desktops) are unreachable, don't know why, we got the deployment agent services restarted there even. Can we stop the Deployment Agents in client tiers, will it have a negative impact on accessing the SAS applications from there?

Java heap size on SASServer1_1 has already been increased from 4 GB to 8 GB, but it did not help.

[gwootton]

Stopping the deployment agents on the client machines shouldn't have an impact on those clients. I would guess they are unreachable because their port (5660) is being blocked by a firewall, so I'm not sure that stopping them would prevent those packets from being dropped.

[sandeeppajni1]

Hi @gwootton,

That seems to be a good information to verify if the 5660 ports is somehow getting blocked due to which server machines (middle tier) are not able to reach out to the client machines for SAS Deployment Backup. Is only 5660 port used by the SAS Deployment or are there any other ports I should be checking on the client machines that must be listening?

[gwootton]

I think it's just 5660, you can confirm the port in use in your <SASHome>\SASDeploymentAgent\9.4\config\agent.properties file.

Designating Ports and Multicast Addresses
https://go.documentation.sas.com/doc/en/bicdc/9.4/biig/n02027intelplatform00install.htm