topic Re: SAS creating orphan jobs/processes when a SAS program has some error, taking server resources in Administration and Deployment

SAS creating orphan jobs/processes when a SAS program has some error, taking server resources

sandeeppajni2 — Fri, 11 Mar 2022 07:13:49 GMT

Hi,

We have been using SAS 9.4 M6 where were have a cluster of 3 Meta nodes, 6 Compute nodes and 2 Web nodes. All these servers are on Linux OS.

We have been facing below two issues, which seem to be related, and we are looking for solution to these problems.

1. Whenever a SAS program has some error in it, the programmers close EG and create a new session or SAS EG closes itself, but the process IDs related to the session keep on running in the background and become orphan. Even if the faulty program is run through command line, it behaves the same, gives an error but the orphan process ID keeps on running at the background. Such orphan processes keep using the resources and take CPU usage to 100% or more. How to resolve such a problem?

2. Due to the high usage of the server resources, the particular compute node stops taking any new load, which is understandable. However, the whole Grid stops taking any new load or user session even within rest of the compute nodes for that particular App server. This makes the whole Grid hung in distributing the load and until we restart the Spawner in that particular node where process ID has reached to 100%, no new sessions are connected. So, we are unable to understand why the SAS Grid is not distributing the load to other compute nodes which are working fine.

Can someone please help with providing solution to these problems. Sometimes, if a particular Grid node reached to 100% CPU usage for user process IDs has also the WIP services running in it, that affects the whole SAS Studio access as well.

Thanks

Re: SAS creating orphan jobs/processes when a SAS program has some error, taking server resources

SASKiwi — Fri, 11 Mar 2022 07:31:14 GMT

When you close EG normally it should clear any SAS sessions associated with it. Even if there have been errors and close EG normally (File-Close etc.) all SAS sessions should close. The only time I've seen orphan SAS sessions remain is if EG hangs or "wheel spins" and you have to kill it with Task Manager.

If this is not the behaviour you are seeing then I suggest to open a track with SAS Tech Support as the causes for this abnormal behaviour will require further investigation.

It is reasonably common practice to schedule the stopping and restarting of SAS server services on a regular basis, say daily or weekly and this will drop any orphan processes. You could perhaps use OS commands (Unix kill command) instead to remove any SAS sessions that run longer than a particular limit, like say 24 hours.

Re: SAS creating orphan jobs/processes when a SAS program has some error, taking server resources

Kurt_Bremser — Fri, 11 Mar 2022 07:36:32 GMT

Start by educating your users to not simply shut down EG, but issue a "Cancel" for the submitted code.

If a program started from the commandline goes into excessive CPU usage, it must have some faulty code in it that causes an infinite loop; this has to be corrected in the code. Any ERROR would simply cause the SAS session to go into syntax check mode and terminate rather quickly.

If you find a condition that causes an infinite loop without the SAS code being faulty, get in contact with SAS Technical Support, as this would be a bug in the SAS software itself.

I have introduced a measure against excessive usage by users. In the WorkspaceServer_usermods.sh shell script, I added code that checks for the number of processes already active for a given user. If this exceeds a threshold, the script exits right there. This means that users with crashed sessions have to contact the SAS admin who can take care of the orphaned processes.

As a SAS admin, it is part of your job to keep an eye on the state of your server(s). You might want to have a script running on your servers which detects overload conditions and sends you an email or other notification.

Re: SAS creating orphan jobs/processes when a SAS program has some error, taking server resources

sandeeppajni2 — Fri, 11 Mar 2022 07:46:47 GMT

Hi @SASKiwi

Thanks for your quick reply. All the suggestions you have provided regarding monitoring of the server resources are already in place, like weekly reboot, killing of any jobs running for more than 24 hours (script scheduled to run every 12 hours). However, as soon as such orphan processes are created, they straight away start using the server resources and make the CPU usage to 100% or more, the SAS hangs then. We are in discussions with the infra team to also prepare some custom scripts which can identify such user processes taking 100% CPU, however it looks like a little difficult for them to identify each such user processes. They have measures for the complete server monitoring, but not for each user processes. Any suggestions to monitor such processes as well would help.

Re: SAS creating orphan jobs/processes when a SAS program has some error, taking server resources

Kurt_Bremser — Fri, 11 Mar 2022 12:08:36 GMT

Activate server-side logging for the workspace server to see what kind of codes run when the issue happens. Maybe you can identify a common event that causes the abnormal CPU usage.

Re: SAS creating orphan jobs/processes when a SAS program has some error, taking server resources

SASKiwi — Sat, 12 Mar 2022 00:52:16 GMT

@sandeeppajni2 - Again I'd suggest engaging SAS Tech Support to help with further diagnosing. Perhaps you have a SAS setting or configuration issue that is contributing to this. If you are maxing out on CPU due to orphan SAS processes within a 24-hour period then I'd say something is drastically going wrong somewhere.

It would also be useful to run a series of tests with EG to identify what EG behaviours result in orphan processes. For example, closing EG normally versus abnormally, killing while programs are still running. Is EG the only source of the problem?

Re: SAS creating orphan jobs/processes when a SAS program has some error, taking server resources

sandeeppajni2 — Sat, 12 Mar 2022 19:50:52 GMT

Hi @SASKiwi

We have observed SAS behaves the same when run from command line, i.e. if there is an error in the SAS program, then also orphan processes are getting created in the background which take server resources.

Re: SAS creating orphan jobs/processes when a SAS program has some error, taking server resources

SASKiwi — Sat, 12 Mar 2022 22:44:03 GMT

@sandeeppajni2 - This is definitely not normal SAS behaviour. Is it possible you have the SAS system option ERRORABEND set on one grid node but not on another? I have no experience with SAS Grid, but I would imagine if you don't align SAS system options across all nodes that may cause problems. Again Tech Support is your best bet to diagnose what is causing this.

Re: SAS creating orphan jobs/processes when a SAS program has some error, taking server resources

Kurt_Bremser — Sun, 13 Mar 2022 05:57:29 GMT

Please show us the error messsages you find.

Re: SAS creating orphan jobs/processes when a SAS program has some error, taking server resources

gwootton — Mon, 14 Mar 2022 14:17:00 GMT

Under normal operation when a Workspace Server has no clients the Object Spawner will terminate it. It sounds like these may be cases where users have submitted code triggering an infinite loop or something similar. I would agree that regularly restarting your environment can clear these out. You can also configure your grid with resource limits to prevent jobs from running longer than a given amount of time actively or idle. In that case the grid will terminate the job after it exceeds that value.

Re: SAS creating orphan jobs/processes when a SAS program has some error, taking server resources

Jlochoa — Mon, 14 Mar 2022 15:12:16 GMT

I created a bat file that uses Putty to log the SAS user into the Linux server to run a bash file to kill zombie SAS sessions with the username. Then run the SAS cleanwork utility.

Re: SAS creating orphan jobs/processes when a SAS program has some error, taking server resources

AhmedAl_Attar — Mon, 14 Mar 2022 15:48:28 GMT

Hi @sandeeppajni2

You have multiple levels of controlling this behavior from the SAS Server side

At the SASApp context level:
- Modify and apply your site specific settings to the [SAS Config]/Lev1/SASApp/sasv9_usermods.cfg file, by adding -ERRORABEND to file
At the Server Context level (BatchServer, WorkspaceServer, ...etc)
- Modify and apply your site specific settings to the [SAS Config]/Lev1/SASApp/WorkspaceServer/sasv9_usermods.cfg file, by adding -ERRORABEND to file

Hope this helps

Re: SAS creating orphan jobs/processes when a SAS program has some error, taking server resources

sandeeppajni2 — Mon, 14 Mar 2022 17:07:37 GMT

Hi @gwootton,

We are already monitoring and have kept OS level scripts in place to auto kill the long running jobs. However, our concern is why SAS Grid is failing to transfer any new requests to a new Compute node with 100% efficiency. We have analyzed if the orphan processes are created, they start taking most of the resources of that server, and in case if new SAS connection requests keep on coming, the SAS Grid algorithm fails and do not distribute jobs among other compute nodes until we restart the Spawner session in that particular node. So, we end up in end users raising connection issues to the Workspace servers and then we validate each of the nodes to identify the faulty one for restarting the Spawner in it.

Re: SAS creating orphan jobs/processes when a SAS program has some error, taking server resources

sandeeppajni2 — Mon, 14 Mar 2022 17:08:37 GMT

Hi @Jlochoa

Any basic idea that you can give on how to get a node scanned for such zombie SAS sessions with the username?

Re: SAS creating orphan jobs/processes when a SAS program has some error, taking server resources

sandeeppajni2 — Mon, 14 Mar 2022 17:11:17 GMT

Hi @Kurt_Bremser

It's not an error that we see. We just see some end user named sas sessions taking 100% CPU or more in a particular compute node. After Grid stops taking new load, we identify the faulty node having issues by validating each node and wherever we see connection problems we just restart the Spawner services of that particular node's App server. This fixes the issue.

Re: SAS creating orphan jobs/processes when a SAS program has some error, taking server resources

sandeeppajni2 — Mon, 14 Mar 2022 17:13:19 GMT

Hi @AhmedAl_Attar

That is something we would not want to limit end users to have their sessions killed. Even the SAS session is killed the background jobs are kept running, eventually taking all the resources in that particular compute server node. So, we end up with the second problem that I described

Re: SAS creating orphan jobs/processes when a SAS program has some error, taking server resources

gwootton — Mon, 14 Mar 2022 17:15:16 GMT

When a job is submitted to the grid, the grid will put the job in a pending state while it makes a determination where to send the job for execution. If new jobs are not being executed anywhere that suggests that there are no eligible hosts for that job. This could be because the nodes have all exceeded their configured thresholds (i.e. the nodes are all busy), have no available job slots (i.e. the nodes are full) or the nodes have been administratively closed. It's also possible to limit the number of jobs a single user can submit to a queue or host, so it could be a user level limitation.
Stopping the spawner would cause the spawner to terminate any jobs it submitted to the grid, which could clear up those type of issues. You would need to examine the grid hosts listing to determine why a given host is not accepting new jobs.

Re: SAS creating orphan jobs/processes when a SAS program has some error, taking server resources

AhmedAl_Attar — Mon, 14 Mar 2022 17:32:06 GMT

Can you please explain how these Background jobs are kept alive if the main SAS server session that running these jobs is terminated?
The second question that comes to mind -- Are you talking about background SAS jobs or Linux Bash jobs?

Re: SAS creating orphan jobs/processes when a SAS program has some error, taking server resources

sandeeppajni2 — Mon, 14 Mar 2022 17:41:05 GMT

Hi @gwootton

I would assume just the opposite that as soon as we restart the Spawner service of that particular node's App server, Grid is able to work freely to start distributing the jobs again. In our case the other nodes are available with the slots but Grid is somehow stuck with filling the load in that particular node itself where usage is 100% and does transfer the load the other nodes automatically for distributing the new session requests. Is it a normal behavior of the system?

Re: SAS creating orphan jobs/processes when a SAS program has some error, taking server resources

sandeeppajni2 — Mon, 14 Mar 2022 17:44:21 GMT

Hi @AhmedAl_Attar

It's the user's process id in the OS we see running even if the program running through command line or through EG which has errors in it is shown closed or terminated.