topic Re: Size of files like SAS_work* increasing exponentially in /saswork in Administration and Deployment

Size of files like SAS_work* increasing exponentially in /saswork

amitvermajhs — Fri, 18 Nov 2016 06:59:20 GMT

Hi ,

I am getting exponential increase in the size of SAS_work* files in my environment which is causing the mount of 1.5TB to fill in a day time. Please guide me how to check which process is taking the maximum space.

One more confirmation, if I close my current session of SAS EG, should the files SAS_work* should delete by itself, or they needs to be deleted manually.

Regards

Amit

Re: Size of files like SAS_work* increasing exponentially in /saswork

Kurt_Bremser — Fri, 18 Nov 2016 12:37:18 GMT

When a SAS process terminates gracefully, the work directories should be deleted. If processes "hang" in a never-ending step, they may keep running even though their client connection is lost. In that case the process(es) need to be killed by operating system means.

Run the cleanwork utility on your work location, and see if that removes the space-eating directories (cleanwork removes directories that have no running process). If not, look for SAS processes that should be forcibly terminated.

Re: Size of files like SAS_work* increasing exponentially in /saswork

amitvermajhs — Fri, 18 Nov 2016 09:07:05 GMT

Can we run the cleanwork utility from SAS EG, or deleting the sessions from UNIX is the only options that we have.

Re: Size of files like SAS_work* increasing exponentially in /saswork

Kurt_Bremser — Fri, 18 Nov 2016 09:25:20 GMT

@amitvermajhs wrote:

Can we run the cleanwork utility from SAS EG, or deleting the sessions from UNIX is the only options that we have.

Since the cleanwork utility has to remove whole directory trees belonging to other users, it has to be run with superuser privileges. That usually precludes running it from EG.

I suggest having a cron job installed that runs cleanwork regularly (eg every hour). This may have to be done by the system admin, or whoever has access to the root account.

The utility can be found as

!SASROOT/SASFoundation/9.4/utilities/bin/cleanwork

(replace !SASROOT with the path to your SAS install directory)

Searching for "dead" workspace servers can also be accomplished from the commandline. I use this command:

ps -ef|grep bridge|grep spawned|sort

If you correlate the output with the sessions reported by your users, you'll find processes that can be killed (again, superuser privileges required).

cleanwork can then take care of the work directories.

Now, you could circumvent the superuser privilege problem by having the necessary commands put into shell scripts, and run those with sudo. Your system admin can use this tool to temporarily grant a "normal" user access to root for certain operations. This might make it possible to run your commands from EG (if XCMD is enabled).

Re: Size of files like SAS_work* increasing exponentially in /saswork

MargaretC — Fri, 18 Nov 2016 13:47:50 GMT

What version of SAS are you using? If SAS 9.4, you can use SAS Environment Manager to track the usage of SAS WORK by userID.
If not, you can use operating system commands to capture information about the file system usage every xx minutes and write this information to a file. You can write a SAS job to read in the file and gather the information.

Re: Size of files like SAS_work* increasing exponentially in /saswork

JuanS_OCS — Fri, 18 Nov 2016 16:20:43 GMT

Hello @MargaretC, I heard a couple of times that part about the EVM to track SASWork by user. Could you please pin point to the right information/documentation to do this? Thank you.

Re: Size of files like SAS_work* increasing exponentially in /saswork

amitvermajhs — Wed, 23 Nov 2016 07:48:19 GMT

Hi,

Just wanted to confirm that which applications use the /saswork for operations, as we have SAS EG , SAS Eminer , SAS CI in our enviroment.

As our 2TB size of /saswork is getting filled in our single day. The highest dataset size is 0.336 TB, total data size is approx 4 TB and we have 4 concurrent users.

How can we check the process which process and applications are using the space in /saswork.

Re: Size of files like SAS_work* increasing exponentially in /saswork

Kurt_Bremser — Wed, 23 Nov 2016 07:56:35 GMT

You can infer the process number from the directory name. The process number is hexadecimally coded into the directory name.

Then do a

ps -fp processnumber

to determine the user of the process.

Or do

ps -ef|grep processnumber

so you get the complete commandline of the process, which will help you determining what type of SAS process you have.

I don't know how good SAS Applications like Eminer or CI are at housekeeping in their work's.

With EG it's up to the users to do that. With a quota system you can enforce that.

Re: Size of files like SAS_work* increasing exponentially in /saswork

amitvermajhs — Wed, 23 Nov 2016 08:03:28 GMT

Thanks for the prompt reply.

Just wanted to confirm that does application like SAS EMiner and SAS CI uses /saswork for the operations.

Regards

Amit Verma

Re: Size of files like SAS_work* increasing exponentially in /saswork

Kurt_Bremser — Wed, 23 Nov 2016 08:09:04 GMT

Well, that depends on the configuration. SAS has different configs for different server types (take a deep look at the Lev1 subtree in your SAS configuration tree, and at the command defintions in the server metadata), which implies that you can set -WORK and -UTILLOC individually. One could even make those locations dependent on user IDs or other conditions.

Since we only use SAS EBI Server (EG, OLAP, STP, Batch), I can't really help you with which servers are called by EM or CI.

Re: Size of files like SAS_work* increasing exponentially in /saswork

amitvermajhs — Wed, 23 Nov 2016 10:37:57 GMT

Hi,

If I say that I have a single user i.e. sasdemo in my environment , and many people are using the same for operations.

And if I waana check that how many concurrent sessions are being opened, then can we get that by

ps -ef|grep sasdemo

or there are others ways of doing so.

Regards

Amit

Re: Size of files like SAS_work* increasing exponentially in /saswork

amitvermajhs — Wed, 23 Nov 2016 10:53:18 GMT

Hi ,
We are getting files like SAS_workEE5c00004F71_servername being created in /saswork mount point, and the size of such file is increasing exponentially. How can I track which process is using it and other related information.

Regards
Amit

Re: Size of files like SAS_work* increasing exponentially in /saswork

Kurt_Bremser — Wed, 23 Nov 2016 11:32:58 GMT

@amitvermajhs wrote:
Hi ,
We are getting files like SAS_workEE5c00004F71_servername being created in /saswork mount point, and the size of such file is increasing exponentially. How can I track which process is using it and other related information.

Regards
Amit

The last part of the hex number right before the _servername contains the process number. In your case, 04F71.

That translates to 20337, so

ps -fp 20337|head

will immediately show you the owner of the process, current clock ticks, starting time, CPU used, and the whole command used in starting the process.

(Mind that my command is the one used on AIX, the Linux ps might behave a little different)

Re: Size of files like SAS_work* increasing exponentially in /saswork

Kurt_Bremser — Wed, 23 Nov 2016 11:39:40 GMT

@amitvermajhs wrote:

Hi,

If I say that I have a single user i.e. sasdemo in my environment , and many people are using the same for operations.

And if I waana check that how many concurrent sessions are being opened, then can we get that by

ps -ef|grep sasdemo

or there are others ways of doing so.

Regards

Amit

This command will show you the processes of sasdemo. But you run into the main disadvantage of using a shared account: nobody is responsible, as anybody is responsible. And your auditors will have a lot of fun with you once something untoward happens (leaked data?)

Therefore I strongly discourage such usage in a production environment.

There's a reason why sasdemo is called that.

Re: Size of files like SAS_work* increasing exponentially in /saswork

amitvermajhs — Thu, 24 Nov 2016 07:18:38 GMT

I agree with the disadvantage of using a generic user, but its as per the client recommendation.

We have a generic user ‘sasdemo’ which is being used by multiple users for operations, so if I want to check the number of concurrent sessions been open. Can I use the command

CL12CM:/saswork >ps -ef|grep sasdemo

sasdemo 16097 16057 0 Nov 17 ? 0:00 saselssrv 13 10 10 2 12800e -classfactory 440196D4-90F0-11D0-9F41-00A024BB830C

sasdemo 27866 17364 0 Nov 11 ? 1:01 /sas/SASHOME/SASFoundation/9.4/sasexe/sas -nodms -noterminal -noxcmd -netencryptalgorithm SASProprietary -metaserver cl12md -me

sasdemo 23623 23583 0 Nov 11 ? 0:00 saselssrv 13 10 10 2 fefc6 -classfactory 440196D4-90F0-11D0-9F41-00A024BB830C

sasdemo 23583 17364 0 Nov 11 ? 1:40 /sas/SASHOME/SASFoundation/9.4/sasexe/sas -nodms -noterminal -noxcmd -netencryptalgorithm SASProprietary -metaserver cl12md -me

sasdemo 27693 17364 0 Nov 11 ? 1:04 /sas/SASHOME/SASFoundation/9.4/sasexe/sas -nodms -noterminal -noxcmd -netencryptalgorithm SASProprietary -metaserver cl12md -me

sasdemo 19815 19122 1 11:41:09 pts/2 0:00 grep sasdemo

sasdemo 16057 17364 0 Nov 17 ? 8:08 /sas/SASHOME/SASFoundation/9.4/sasexe/sas -nodms -noterminal -noxcmd -netencryptalgorithm SASProprietary -metaserver cl12md -me

sasdemo 21772 21732 0 Nov 18 ? 0:00 saselssrv 13 10 10 2 fb991 -classfactory 440196D4-90F0-11D0-9F41-00A024BB830C

sasdemo 27906 27866 0 Nov 11 ? 0:00 saselssrv 13 10 10 2 10d198 -classfactory 440196D4-90F0-11D0-9F41-00A024BB830C

sasdemo 16254 17364 0 10:12:47 ? 53:48 /sas/SASHOME/SASFoundation/9.4/sasexe/sas -nodms -noterminal -noxcmd -netencryptalgorithm SASProprietary -metaserver cl12md -me

sasdemo 21732 17364 0 Nov 18 ? 0:32 /sas/SASHOME/SASFoundation/9.4/sasexe/sas -nodms -noterminal -noxcmd -netencryptalgorithm SASProprietary -metaserver cl12md -me

sasdemo 1448 17364 0 Nov 15 ? 1:03 /sas/SASHOME/SASFoundation/9.4/sasexe/sas -nodms -noterminal -noxcmd -netencryptalgorithm SASProprietary -metaserver cl12md -me

sasdemo 27733 27693 0 Nov 11 ? 0:00 saselssrv 13 10 10 2 10d0e4 -classfactory 440196D4-90F0-11D0-9F41-00A024BB830C

sasdemo 19120 19114 0 11:25:29 ? 0:00 sshd: sasdemo@pts/2

sasdemo 19814 19122 4 11:41:09 pts/2 0:00 ps -ef

sasdemo 16298 16254 0 10:12:48 ? 0:00 saselssrv 13 10 10 2 1016ed -classfactory 440196D4-90F0-11D0-9F41-00A024BB830C

sasdemo 1488 1448 0 Nov 15 ? 0:00 saselssrv 13 10 10 2 117999 -classfactory 440196D4-90F0-11D0-9F41-00A024BB830C

sasdemo 19122 19120 1 11:25:31 pts/2 0:00 -sh

root 19114 1241 0 11:25:17 ? 0:00 sshd: sasdemo [priv]

CL12CM:/saswork >

And the /sas/SASHOME/SASFoundation/9.4/sasexe/sas -nodms -noterminal -noxcmd -netencryptalgorithm SASProprietary -metaserver cl12md –me can be regarded as the concurrent sessions, or are there any other way to find the concurrent sessions.

We have directories like below , consuming our major space in /saswork. What is the difference between SAS_work * and SAS_util * files.

439869216 SAS_workEDC100005DC8_server_name

11220960 SAS_work2235000006D4_ server_name

1569216 SAS_util000100005DC8_ server_name

If we delete the directories SAS_work* will they affect our system , other than deleting the process running it.

Regards

Amit

Re: Size of files like SAS_work* increasing exponentially in /saswork

Kurt_Bremser — Thu, 24 Nov 2016 07:51:13 GMT

Have you already run the cleanwork utility on your WORK location?

I ask this because 005DC8 corresponds to decimal 24008, and there is no such process in your list. The same goes for 06D4 (1748).

cleanwork would automatically take care of orphaned directories

The SASutil directories are created parallel to the work directories, and contain the utility files of sort and other procedures; their physical location can be controlled by the -utilloc system option.

As such utility files are usually automatically deleted once the corresponding procedure finishes, a permanently non-empty SASutil is a sure sign of a crashed process.

Re: Size of files like SAS_work* increasing exponentially in /saswork

amitvermajhs — Thu, 24 Nov 2016 09:29:53 GMT

My IT team has deleted the SAS_work.* files from the system as a temporary solution. That is what I wanted to confirm, will that have any negative effect on my system.

Re: Size of files like SAS_work* increasing exponentially in /saswork

amitvermajhs — Thu, 24 Nov 2016 09:32:08 GMT

Please do confirm, that the process that I have followed for checking the parallel sessions is correct, or is their alternate way of checking the parallel sessions.
As I need to check that we have 4 concurrent users and how many sessions are active, to check if their are others users to my system.

Re: Size of files like SAS_work* increasing exponentially in /saswork

Kurt_Bremser — Thu, 24 Nov 2016 09:41:38 GMT

I STRONGLY suggest you use the cleanwork utility, as any "manual" deletions carry the risk of pulling data accidentally out from "under" a still running process.

As for how to check which directory belongs to which process, I have already given you the information.

Re: Size of files like SAS_work* increasing exponentially in /saswork

amitvermajhs — Thu, 24 Nov 2016 09:53:42 GMT

I tried converting the hexadecimal to decimal for all the process, that are running in my system. But the ps -ef|grep process_number is producing no result.