08-20-2015 09:53 PM
We are using Solaris machine for our SAS.
and I found suddenly File system /sas, /sas/saswork, /sas/sasutil reached 100%.not sure how suddenly this happened.
anyone have any idea how can i check which process or job or which user has occupied more space and that made the file system 100%.
which command I need to use,I have used several commands like du-a,du -s,du -hd,du -k, du -a /sas | sort -n -r | head -n 10...etc
but these did not help me.
I want to find which job or which user occupied more space because of which suddenly the space reached 100%.
I would really appreciate if any one can provide your valuable suggestion which commands i should use and where i need to check
Thanks in advance
08-20-2015 10:15 PM
Run a file listing on sas/work and check the owners of the largest files or who owns the largest portion. It may not be one user, it could be having 3 users running large jobs at the same time. Or the space has not been cleaned up in a while and there lots of stuff lying around.
08-20-2015 10:55 PM
I wouldn't necessarily assume that it is one user / job that has filled up SASWORK. Look at the number of of _TDnnnn folders in SASWORKIf there are a lot and the dates they are created are spread over time it could be that these are just SAS jobs/sessions that did not properly clear their WORK space.
Many SAS sites run the SAS CLEANUP process daily to deal with this very issue.
08-20-2015 11:34 PM
Thanks for your prompt replies
Till evening the space was at 30 % and suddenly it reached above 90% and slowly within couple of hours it reached 100 %.
and suddenly after some time all the space came to normal 30 %.
/sas is the main mounting point on which all other mounting points falls as below
when I watched all these locations were at 90 % and slowly they reached 100%.
so In this kind of situation what is the immediate step i need to take and how can i find which files or users occupying more space and how can i resolve this issue immediately.
and what are the file systems i need to check
/saswork location i did'nt find any files with the name _TDnnnn folders and i did'nt find any files with bigger than in mega bytes
I want to find exactly how can i find which file system and job or user occupied more space which has increased space suddenly to maximum
08-21-2015 12:07 AM
It is not clear from your response whether it is /saswork that is the problem or not. Please clarify.
Every SAS job will create a directory under SASWORK like /saswork/_TDnnnn_servername_ If there aren't any directories like this then where is SAS writing its WORK files?
In a SAS session check the properties of the WORK libname to confirm your WORK location.
08-21-2015 02:54 AM
For shared directories (one of them being the SAS work location, the other /home, where the sasuser libs are usually located), quota management for the users is a must. And I mean MUST.
Quota management will mean that a user with an ill-formed request will experience an individual loss of service, without denying the service to all others.
Once you have a near-to-full or full condition, I would run something like
find /sas -type f -ls
in a filename pipe and read the output into a SAS dataset. From that, you can use your SAS skills to find the offending file(s).
Be aware that files without a directory entry (but that still use disk space) may exist in UNIX, they can be found by executing
fuser -dcV path
where path must be in the filesystem (not just path) in question. Gives you a list of processes that hold a handle on a completely unlinked file, if such exist.
08-22-2015 07:57 PM
It could be that a user run some "insane" job, let's say something like a cartesian product between two very big tables. That could fill-up saswork & sasutil quite a bit. And once the session finishes all the space would get freed-up almost immediately.
What "disturbs" me a bit: You say that space usage "jumped" from 30% to 90% in almost no time. Assuming your disk space is considerable writing to disk filling up 60% of space should take some time. So I'm asking myself if there had been some infrastructure work been going on in the background and you simply didn't always have the same amount of disk space available. I would have a chat with your UNIX system admin to find out what happened.
08-24-2015 02:25 AM
In this kind of situation, there should be at least one user complaining about failing jobs/steps. How many users are working on your installation? Maybe just asking all users if they have problems could reveal the source of the disk full condition.
08-24-2015 05:14 AM
Its on Development environment and we have very few users working may be 4 or 5 users on this and no one has complained about failing jobs the space is going back to normal immediately in few mins after it touches 100%.
08-24-2015 06:18 AM
Then I'd write a shell script that checks for a certain usage level, and if a treshold is reached, executes a du -sk in the filesystem, appends the result to a file, and notifies you. Run this script every minute using cron, and, once you have identified where the large files are placed, modify the script to dig deeper. Sooner or later you will find the file(s), and the user responsible.
08-21-2015 06:34 AM
When your users are of the type analytics miners, then do not use any quotas you can isolate them on a dedicated saswork filesystem.
I am against quotas because you are going into a fight against users needing to do their work. When hurring them they will look for other ways to achieve their goals.
One side of the company is keeping the other side busy without having any business efficiency (job creation).
When you have predictable workload that is have verified and tested sas programs running. Their behavior is mostly planning. Only not noticed program bugs will cause errors. The user having problems will complain about those.
08-21-2015 06:50 AM
Well, it's my job to identify the needs of users and set their quotas accordingly. Powerusers are assigned to a different class and get additional workspaces, but I still prevent them from clogging up resources that they share with other users.