topic File system /sas ,/sas/saswork /sas/sasutil reached 100% in SAS Enterprise Guide

File system /sas ,/sas/saswork /sas/sasutil reached 100%

sasprofile — Fri, 21 Aug 2015 01:53:09 GMT

Hi Friends.

We are using Solaris machine for our SAS.

and I found suddenly File system /sas, /sas/saswork, /sas/sasutil reached 100%.not sure how suddenly this happened.

anyone have any idea how can i check which process or job or which user has occupied more space and that made the file system 100%.

which command I need to use,I have used several commands like du-a,du -s,du -hd,du -k, du -a /sas | sort -n -r | head -n 10...etc

but these did not help me.

I want to find which job or which user occupied more space because of which suddenly the space reached 100%.

I would really appreciate if any one can provide your valuable suggestion which commands i should use and where i need to check

Thanks in advance

Re: File system /sas ,/sas/saswork /sas/sasutil reached 100%

Reeza — Fri, 21 Aug 2015 02:15:18 GMT

Run a file listing on sas/work and check the owners of the largest files or who owns the largest portion. It may not be one user, it could be having 3 users running large jobs at the same time. Or the space has not been cleaned up in a while and there lots of stuff lying around.

Re: File system /sas ,/sas/saswork /sas/sasutil reached 100%

SASKiwi — Fri, 21 Aug 2015 02:55:19 GMT

I wouldn't necessarily assume that it is one user / job that has filled up SASWORK. Look at the number of of _TDnnnn folders in SASWORKIf there are a lot and the dates they are created are spread over time it could be that these are just SAS jobs/sessions that did not properly clear their WORK space.

Many SAS sites run the SAS CLEANUP process daily to deal with this very issue.

Re: File system /sas ,/sas/saswork /sas/sasutil reached 100%

sasprofile — Fri, 21 Aug 2015 03:34:24 GMT

Thanks for your prompt replies

Till evening the space was at 30 % and suddenly it reached above 90% and slowly within couple of hours it reached 100 %.

and suddenly after some time all the space came to normal 30 %.

/sas is the main mounting point on which all other mounting points falls as below

/sas/guide

/sas/saswork

/sas/util

when I watched all these locations were at 90 % and slowly they reached 100%.

so In this kind of situation what is the immediate step i need to take and how can i find which files or users occupying more space and how can i resolve this issue immediately.

and what are the file systems i need to check

/saswork location i did'nt find any files with the name _TDnnnn folders and i did'nt find any files with bigger than in mega bytes

I want to find exactly how can i find which file system and job or user occupied more space which has increased space suddenly to maximum

Re: File system /sas ,/sas/saswork /sas/sasutil reached 100%

SASKiwi — Fri, 21 Aug 2015 04:07:48 GMT

It is not clear from your response whether it is /saswork that is the problem or not. Please clarify.

Every SAS job will create a directory under SASWORK like /saswork/_TDnnnn_servername_ If there aren't any directories like this then where is SAS writing its WORK files?

In a SAS session check the properties of the WORK libname to confirm your WORK location.

Re: File system /sas ,/sas/saswork /sas/sasutil reached 100%

sasprofile — Fri, 21 Aug 2015 04:13:56 GMT

Yes the problem is with all below mounting points which reached 100 %

/sas

/sas/saswork

/sas/sasutil

Re: File system /sas ,/sas/saswork /sas/sasutil reached 100%

LinusH — Fri, 21 Aug 2015 04:53:22 GMT

Talk to your UNIX admin for a way to monitor disk usage per user.

Re: File system /sas ,/sas/saswork /sas/sasutil reached 100%

Kurt_Bremser — Fri, 21 Aug 2015 06:54:17 GMT

For shared directories (one of them being the SAS work location, the other /home, where the sasuser libs are usually located), quota management for the users is a must. And I mean MUST.

Quota management will mean that a user with an ill-formed request will experience an individual loss of service, without denying the service to all others.

Once you have a near-to-full or full condition, I would run something like

find /sas -type f -ls

in a filename pipe and read the output into a SAS dataset. From that, you can use your SAS skills to find the offending file(s).

Be aware that files without a directory entry (but that still use disk space) may exist in UNIX, they can be found by executing

fuser -dcV path

where path must be in the filesystem (not just path) in question. Gives you a list of processes that hold a handle on a completely unlinked file, if such exist.

Re: File system /sas ,/sas/saswork /sas/sasutil reached 100%

jakarman — Fri, 21 Aug 2015 10:34:03 GMT

When your users are of the type analytics miners, then do not use any quotas you can isolate them on a dedicated saswork filesystem.
I am against quotas because you are going into a fight against users needing to do their work. When hurring them they will look for other ways to achieve their goals.

One side of the company is keeping the other side busy without having any business efficiency (job creation).

When you have predictable workload that is have verified and tested sas programs running. Their behavior is mostly planning. Only not noticed program bugs will cause errors. The user having problems will complain about those.

Re: File system /sas ,/sas/saswork /sas/sasutil reached 100%

Kurt_Bremser — Fri, 21 Aug 2015 10:50:11 GMT

Well, it's my job to identify the needs of users and set their quotas accordingly. Powerusers are assigned to a different class and get additional workspaces, but I still prevent them from clogging up resources that they share with other users.

Re: File system /sas ,/sas/saswork /sas/sasutil reached 100%

AnandVyas — Fri, 21 Aug 2015 11:04:54 GMT

Hi,

Try to search for the files that are consuming the large space under the mount point /sas
using the command:

find /sas -size +5MB ( you can change the size as per your requirement)

Once you have them listed, just check out if there are any recurring files that are causing this and do some housekeeping to remove them.

Incase they are getting generated again, then dig in to see which process is creating them and why.

In 9.4 release, there is a known issue where lot of SAS Web server logs gets generated and needs to be removed. (Just Incase you are using SAS Webserver)

Thanks!

Re: File system /sas ,/sas/saswork /sas/sasutil reached 100%

sasprofile — Fri, 21 Aug 2015 19:05:47 GMT

Again we got space issues on our Server today morning, all of sudden sometimes the file system sas/saswork is
reaching Max 100 % suddenly and there by all other mounting points are reaching 100%.
We are using Solaris 5.10

And immediately after it reached 100% and the spaces are clearing and going back to normal.
Today morning it happened couple of times /saswork reached max 100% and which impacted other sas mounting points to reach 100%
and cleared space on its own and again started slowly growing and reached 100% and again back to normal.

whatever is causing the space to peak is happening very quickly and the critical condition is upon us by the time we get the alert.

I have checked the saswork directory and I can see only 2 users were working on the server at that time and they are not running any bigger jobs. And also there are no new datasets created recently that might have effected.

I want to know in this situation what would be our immediate action, how can I check which job/process or user
Is occupying more space and how can we bring it to normal.
Which are all the file location we need to check for and how to check using commands ?

So not sure why its behaving crazy.

and I want daily report on files in the space sorted by size and then one sorted by last date accessed?
I want to creatin a space management report in SAS. How can we do that ?

We would really appreciate if anyone can help me with this

Re: File system /sas ,/sas/saswork /sas/sasutil reached 100%

Reeza — Fri, 21 Aug 2015 19:07:48 GMT

sasprofile wrote:

reaching Max 100 % suddenly and there by all other mounting points are reaching 100%.
We are using Solaris 5.10

And immediately after it reached 100% and the spaces are clearing and going back to normal.

and I want daily report on files in the space sorted by size and then one sorted by last date accessed?
I want to creatin a space management report in SAS. How can we do that ?

We would really appreciate if anyone can help me with this

If it clears right after it reaches 100% how is a daily report going to help?

Re: File system /sas ,/sas/saswork /sas/sasutil reached 100%

sasprofile — Fri, 21 Aug 2015 19:12:15 GMT

It did not happened before reaching suddenly 100% and clearing immediately on its own, this has happened for the first time.

can you help me with this Daily report please

Re: File system /sas ,/sas/saswork /sas/sasutil reached 100%

Reeza — Fri, 21 Aug 2015 19:38:19 GMT

This is what I use on my UNIX box, have no idea what the corresponding would be on Solaris:

*Use Unix commands to read the directory information;

filename lsinfo pipe 'cd /folder1/folder2; ls -l';

data work_space;

infile lsinfo pad end=last;

input rec $250.;

dsn=scan(rec,-1,' ');

owner=scan(rec,3,' ');

permissions=scan(rec,1,' ');

size=scan(rec, 5, ' ');

run;

Re: File system /sas ,/sas/saswork /sas/sasutil reached 100%

jakarman — Fri, 21 Aug 2015 19:49:31 GMT

reeza, your are right. Normally you can have logging like nmon/topas being active making snapshots. Mostly every 5 minutes so you can see afterwards what is happening. What sasprofle is saying all mountpoints are filled up at the same moment. The mountpoints/filesystems are visible with the "df -g" command.

saswork and several others should be isolated. It doesn't make sense when all of them get filled up on the same moment.

Having a 200Gb or 500Gb of saswork you can easily seen that growing/filling up as IO speed is not that fast it is capable to do that very quickly it will take as an hour.

The last part of filling up for 80 to 100% will slow down dramatically as the filesystem is needing harder work to find free space.

If the behavior is different something else could be causing that.

Corruptions in the SAN an filesystem could happen and cause this kind of OS system failures. Yup I have experienced those on this kind of machines. All hardware and OS have their failures and at some level lack of reliability. One of those was a lost disk - unknown reason another was failure in DAC rights processing, never found the cause.

Re: File system /sas ,/sas/saswork /sas/sasutil reached 100%

sasprofile — Fri, 21 Aug 2015 20:22:22 GMT

Reeza thanks for the valuable information

am completely new to the Unix Solaris, so am not sure how to create Daily report.

so the same code you provided need to run as .sas program on solaris server to get the daily reports

Re: File system /sas ,/sas/saswork /sas/sasutil reached 100%

jakarman — Sat, 22 Aug 2015 06:16:50 GMT

saspronfile, "main mounting point /sas" . "others like /sas/work below" ?? Unix is different to Windows in storage management. This makes no sense.

With Unix you have storage and you can map that to a directory/folder that is a mountpoint.. With Windows you have a drive/storage where you can define directories folders. With a DFS you can present many of those as as single directory structure. That is a bit different way of thinking.

With a HFS you have those mounpoints and the security inheritance by all level. this is why there is a standardized structure like General overview of the Linux file system

- /opt for third party software eg SAS (with rpm also bin)

- /var for temporary data eg saswork and logging.

- /home for personal "my documents"

- ....

SAS institiute did made a complete mess of this common approach and guidelines by ignoring those. This is commonly causing a lot of trouble needing cooperation by unix admins. You are running out of space/resources. The firtst one is what are the mountpoints.... "df -g" and how big it he work/util?

Re: File system /sas ,/sas/saswork /sas/sasutil reached 100%

Patrick — Sat, 22 Aug 2015 23:57:02 GMT

It could be that a user run some "insane" job, let's say something like a cartesian product between two very big tables. That could fill-up saswork & sasutil quite a bit. And once the session finishes all the space would get freed-up almost immediately.

What "disturbs" me a bit: You say that space usage "jumped" from 30% to 90% in almost no time. Assuming your disk space is considerable writing to disk filling up 60% of space should take some time. So I'm asking myself if there had been some infrastructure work been going on in the background and you simply didn't always have the same amount of disk space available. I would have a chat with your UNIX system admin to find out what happened.

Re: File system /sas ,/sas/saswork /sas/sasutil reached 100%

sasprofile — Sun, 23 Aug 2015 23:19:46 GMT

The space has not jumped immediately from 30% to 100%,it slowly increasing % by % within 1/2 hrs it went to 100%.