topic Re: Kill Hung Sessions Automatically in Administration and Deployment

Kill Hung Sessions Automatically

five — Mon, 06 Dec 2021 14:45:50 GMT

Yesterday we had a user reach out that was able to login to our Viya environment ok, but could not submit code. She would click run and then nothing would hang. I logged in and verified services a status check and then healthcheck and everything seemed good. I wasn't really sure what to do, if it was simply a problem with her code or something else. I then responded and asked her to run PROC SETINIT; RUN; She said that would run the first time she logged into a new session, but if she ran it a second time it would do the same thing. Her mentioning sessions made me wonder how many were open. I did a ps -aux and grepped for her user id and sure enough there was an old session from December 2 still open. I killed the two pid's from back then. Had her run her sas code again and everything was working again.

Anyone have any automated or better fixes for that type of scenario?

Re: Kill Hung Sessions Automatically

Kurt_Bremser — Mon, 06 Dec 2021 17:56:58 GMT

Start with a clear definition of "what constitutes a hung session on the operating system level". You may find that this is not as clear-cut as it looks at first.

One thing that is easy: kill all remaining sessions at a given time during the night.

Re: Kill Hung Sessions Automatically

five — Mon, 06 Dec 2021 19:01:18 GMT

Yea, I am no expert. Why I was asking the community if anyone had an idea of a good way to remedy this.

I have come up with this code, that I am thinking about running once an hour:

ps -eo pid,etimes,comm,group | awk '{if ($3 ~ /compsrv/ && $4 ~ /aduser/ && $2>14400) print $1}'

Returns the process id of our adusers that have compsrv running for more than 4 hours. I don't know if there is a better way to do it. But it's an idea I have at the moment, just to pipe those to kill.

Re: Kill Hung Sessions Automatically

gwootton — Mon, 06 Dec 2021 20:18:19 GMT

Under normal operation I think SAS Studio will terminate the compute session when their session with the user ends, are you seeing a lot of these orphaned sessions out there? I'm concerned terminating any compute server older than 4 hours could result in legitimate sessions being terminated.

Re: Kill Hung Sessions Automatically

five — Mon, 06 Dec 2021 20:28:41 GMT

No, I've only been made aware one time. The user said it had happened before. And the initial email from the professor said multiple students were having the same issue, but I only had an exchange with one student and hers was the only process that was 4 days old.

At the moment I have a few users with sessions greater than 4 hours. I am just going to keep an eye on it and see how it transpires through the week. I could always increase the number to 24 hours or something like that.

Re: Kill Hung Sessions Automatically

AhmedAl_Attar — Tue, 07 Dec 2021 11:10:04 GMT

Hi @five

Please keep in mind killing orphaned processed on their own sometime are not enough, you'll need to cleanup the temporary data these processes leave behind.

In a typical normal SAS process termination, SAS would remove all the temporary files that particular session had created as it terminates/shut downs.

Therefore, for the orphaned process that being manually terminated by user intervention, its temporary files needs to be deleted as well!

Have a look at the SAS cleanwork utility

- Linux

- Windows

Hope this helps,

Ahmed

Re: Kill Hung Sessions Automatically

five — Tue, 07 Dec 2021 12:59:28 GMT

Yea I did run that bash script afterward. If it gives any feedback, when it does something then it must not have done anything.

sudo /opt/sas/spre/home/SASFoundation/utilities/bin/cleanwork /saswork
Running as user root on hostname web01.server.edu

I have run it a few times and it either displays sessions still running or returns nothing like above. And there have been times where ps aux will show sessions still running and there will be nothing listed in /saswork as well.