SAS batch job are long running

[naveenraj]

We have batch jobs that were running fine without any issue for a long time. From last week onwards we are seeing that the jobs are long running. Our's is a sas environment hosted on linux machine. We have not made any changes to the code recently. The only thing i can see within the logs is that the real time of the procedure or data step has increased. We have checked wih Unix admins and they have told that no other process is running on the server. We have also run the jobs at different times to see if its a  timing isue. But did not get any difference. Only thing we can notice is high i/o utilization on sas work library mount which was not there previously. Have anyone faced this issue or guide me on what to check further to imprve the performance

now:

real time 37:04.73
user cpu time 9:49.74
system cpu time 5:39.84
memory 2110359.71k
OS Memory 2120860.00k
Timestamp 07/06/2020 09:43:48 PM
Page Faults 31
Page Reclaims 2235141
Page Swaps 0
Voluntary Context Switches 8831289
Involuntary Context Switches 1730716
Block Input Operations 335352296
Block Output Operations 722203936

before : 

real time 9:30.78
user cpu time 8:15.08
system cpu time 5:14.26
memory 2110359.71k
OS Memory 2120860.00k
Timestamp 06/30/2020 08:40:02 PM
Page Faults 33
Page Reclaims 2068727
Page Swaps 0
Voluntary Context Switches 8136300
Involuntary Context Switches 897939
Block Input Operations 332189320
Block Output Operations 727054200

[Kurt_Bremser]

Since you talk about "work library mount", which lets me suspect that your WORK is possibly a network share (considered VERY BAD PRACTICE), then it means your network connection to that share has somehow degraded.

WORK should either be a very fast internal disk (array), or run over a SAN, with a powerful box (or several, for failover) on the backend.

In fact, I would avoid any SAS processing on network shares; only initial fetches and final stores should be made from/to those.

[naveenraj]

@Kurt_Bremser Apologies on my wording, it is an internal disk to the server 

[Kurt_Bremser]

In your log, the number of involuntary context switches has roughly doubled. This points to possible wait states (causing the SAS process to be kicked out of the CPU), and/or higher priority processes demanding CPU access. Wait states for disk I/O may come from multiple processes doing I/O at the same time.

Log on to the server and run a monitoring utility (nmon, topas, ...) to see the CPU load and other diagnostics.

[SASKiwi]

Also check for any orphan / rogue SAS sessions on your SAS application server and kill them. Often Enterprise Guide / SAS Studio SAS sessions don't close cleanly so orphan SAS sessions processes can build up on your server.

An alternative approach would be to get your whole SAS server platform rebooted if that is allowable.