Hi @gwootton I think you are tying to say which App server to connect to is decided by the end user, however which compute node's Spawner the request goes to is decided in the background. Am I correct? So, on basis of what algorithm or logic the SAS decides to send the request to a specific Object Spawner, that is something I am trying to understand, as somehow SAS decides to send the load to the faulty Spawner or Grid only. I have scheduled some sessions with the SAS SMEs and would try to understand the logic behind the jobs distribution and how it can be optimized in our case. ... View more

Hi @gwootton Thanks for explaining the functioning of the Grid and Spawner processes and their orders of working. So, it is not the Grid which has issues as Spawner itself is not allowing the jobs to go to Grid for distribution (we use SWO/ sgmg). However, I am failing to understand why then after we just restart the particular App server's Spawner service, everything becomes normal (we have 7 SAS App servers). Why all the new jobs are not going to other compute nodes' Spawners but just to that particular faulty Spawner for the users. So, there must be a algorithm or logic running into SAS that is choosing that Spawner only to make the new user connections. I am trying to understand that. Hope I am making a sense. ... View more

Hi @AhmedAl_Attar It's the user's process id in the OS we see running even if the program running through command line or through EG which has errors in it is shown closed or terminated. ... View more

Hi @gwootton I would assume just the opposite that as soon as we restart the Spawner service of that particular node's App server, Grid is able to work freely to start distributing the jobs again. In our case the other nodes are available with the slots but Grid is somehow stuck with filling the load in that particular node itself where usage is 100% and does transfer the load the other nodes automatically for distributing the new session requests. Is it a normal behavior of the system? ... View more

Hi @AhmedAl_Attar That is something we would not want to limit end users to have their sessions killed. Even the SAS session is killed the background jobs are kept running, eventually taking all the resources in that particular compute server node. So, we end up with the second problem that I described ... View more

Hi @Kurt_Bremser It's not an error that we see. We just see some end user named sas sessions taking 100% CPU or more in a particular compute node. After Grid stops taking new load, we identify the faulty node having issues by validating each node and wherever we see connection problems we just restart the Spawner services of that particular node's App server. This fixes the issue. ... View more

Hi @Jlochoa Any basic idea that you can give on how to get a node scanned for such zombie SAS sessions with the username? ... View more

Hi @gwootton, We are already monitoring and have kept OS level scripts in place to auto kill the long running jobs. However, our concern is why SAS Grid is failing to transfer any new requests to a new Compute node with 100% efficiency. We have analyzed if the orphan processes are created, they start taking most of the resources of that server, and in case if new SAS connection requests keep on coming, the SAS Grid algorithm fails and do not distribute jobs among other compute nodes until we restart the Spawner session in that particular node. So, we end up in end users raising connection issues to the Workspace servers and then we validate each of the nodes to identify the faulty one for restarting the Spawner in it. ... View more

Hi @SASKiwi We have observed SAS behaves the same when run from command line, i.e. if there is an error in the SAS program, then also orphan processes are getting created in the background which take server resources. ... View more

Hi @SASKiwi Thanks for your quick reply. All the suggestions you have provided regarding monitoring of the server resources are already in place, like weekly reboot, killing of any jobs running for more than 24 hours (script scheduled to run every 12 hours). However, as soon as such orphan processes are created, they straight away start using the server resources and make the CPU usage to 100% or more, the SAS hangs then. We are in discussions with the infra team to also prepare some custom scripts which can identify such user processes taking 100% CPU, however it looks like a little difficult for them to identify each such user processes. They have measures for the complete server monitoring, but not for each user processes. Any suggestions to monitor such processes as well would help. ... View more

Hi @saspert: Okay. Just to let you know that we recently started facing some problems in accessing SAS Studio, whereas SAS EG was connecting fine, after every instance of complete reboot of SAS services. It turned out to be an issue in the SAS Deployment Backup not completing in time and taking more than usual. So, sometimes the  deployment of web services (SASServer1_1, which plays a major role in running the SAS Studio) was not happening completely after the reboot of the SAS Services. We excluded the backup deployment xml file (SASDeploymentBackup.xml) in the location: ../Lev1/Web/WebAppServer/SASServer1_1/conf/Catalina/localhost, and then the deployment of configurations started happening in time without issues, which fixed the SAS Studio login issue too. It might be helpful to try following these steps in your case too. You need to first stop the SAS Web services, remove the file, and start the services again to do this. ... View more

It was working fine always, used to take not more than 5 mins. However, it started taking hours to start the SASServer1_1 after last month's Linux server patching activity. Although, no issues in start of services even after non-prod environment servers' patching ... View more

Hi @Kurt_Bremser, Could you please explain a bit more about tomcat expanding webapps to/from the staging area. We have been getting the same problem while starting the web apps. It takes so much time (about 2 hours) to start up the services completely and in the Lev1/Web/Logs location the log files are taking so much time to get created, with similar errors: 2021-08-22 08:25:08,940 [org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor#2e9cf871-1] WARN [unknown] com.sas.svcs.cluster.httpinvoker.AutoDiscoveredHttpService - Have received 200 failures while trying to access an auto-discoverable service to satisfy request for 'com.sas.svcs.prefs.client.SASPreferences'. Please ensure that the application and/or process surfacing this interface is still in the process of starting. 2021-08-22 08:30:09,110 [org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor#2e9cf871-1] WARN [unknown] com.sas.svcs.cluster.httpinvoker.AutoDiscoveredHttpService - Have received 220 failures while trying to access an auto-discoverable service to satisfy request for 'com.sas.svcs.prefs.client.SASPreferences'. Please ensure that the application and/or process surfacing this interface is still in the process of starting. 2021-08-22 08:35:09,278 [org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor#2e9cf871-1] WARN [unknown] com.sas.svcs.cluster.httpinvoker.AutoDiscoveredHttpService - Have received 240 failures while trying to access an auto-discoverable service to satisfy request for 'com.sas.svcs.prefs.client.SASPreferences'. Please ensure that the application and/or process surfacing this interface is still in the process of starting. 2021-08-22 08:35:24,278 [org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor#2e9cf871-1] ERROR [unknown] com.sas.svcs.cluster.httpinvoker.AutoDiscoveredHttpService - ############################################################################################# ### Attempting to access service for interface 'com.sas.svcs.prefs.client.SASPreferences' for 1 hours. ##### ### Please ensure that the process providing the service is starting. ####################### ### After 2 hours, the service will be considered unavailable. ############################## ############################################################################################# 2021-08-22 08:40:09,443 [org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor#2e9cf871-1] WARN [unknown] com.sas.svcs.cluster.httpinvoker.AutoDiscoveredHttpService - Have received 260 failures while trying to access an auto-discoverable service to satisfy request for 'com.sas.svcs.prefs.client.SASPreferences'. Please ensure that the application and/or process surfacing this interface is still in the process of starting. 2021-08-22 08:45:09,599 [org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor#2e9cf871-1] WARN [unknown] com.sas.svcs.cluster.httpinvoker.AutoDiscoveredHttpService - Have received 280 failures while trying to access an auto-discoverable service to satisfy request for 'com.sas.svcs.prefs.client.SASPreferences'. Please ensure that the application and/or process surfacing this interface is still in the process of starting. ... View more