File system /sas ,/sas/saswork /sas/sasutil reached 100%

[AnandVyas]

Hi,

Try to search for the files that are consuming the large space under the mount point /sas
using the command:

find /sas -size +5MB ( you can change the size as per your requirement)

Once you have them listed, just check out if there are any recurring files that are causing this and do some housekeeping to remove them.

Incase they are getting generated again, then dig in to see which process is creating them and why.

In 9.4 release, there is a known issue where lot of SAS Web server logs gets generated and needs to be removed. (Just Incase you are using SAS Webserver)

Thanks!

[sasprofile]

Again we got space issues on our Server today morning, all of sudden sometimes the file system sas/saswork is
reaching Max 100 % suddenly and there by all other mounting points are reaching 100%.
We are using Solaris 5.10

And immediately after it reached 100% and the spaces are clearing and going back to normal.
Today  morning it happened couple of times /saswork reached max 100% and which impacted other sas mounting points to reach 100%
and cleared space on its own and again started slowly growing and reached 100% and again back to normal.

whatever is causing the space to peak is happening very quickly and the critical condition is upon us by the time we get the alert. 

I have checked the saswork directory and I can see only 2 users were working on the server at that time and they are not running any bigger jobs. And also there are no new datasets created recently that might have effected.

I want to know in this situation what would be our immediate action, how can I check which job/process or user
Is occupying more space and how can we bring it to normal.
Which are all the file location we need to check for and how to check using commands ?

So not sure why its behaving crazy.

and I want daily report on files in the space sorted by size and then one sorted by last date accessed?
I want to creatin a space management report in SAS. How can we do that ?

We would really appreciate if anyone can help me with this

[Reeza]

sasprofile wrote:

reaching Max 100 % suddenly and there by all other mounting points are reaching 100%.
We are using Solaris 5.10
And immediately after it reached 100% and the spaces are clearing and going back to normal.


and I want daily report on files in the space sorted by size and then one sorted by last date accessed? 
I want to creatin a space management report in SAS. How can we do that ?
We would really appreciate if anyone can help me with this

If it clears right after it reaches 100% how is a daily report going to help?

[sasprofile]

It did not happened before reaching suddenly 100% and clearing immediately on its own, this has happened for the first time.

can you help me with this Daily report please

[Reeza]

This is what I use on my UNIX box, have no idea what the corresponding would be on Solaris:

*Use Unix commands to read the directory information;

filename lsinfo pipe 'cd /folder1/folder2; ls -l';

data work_space;

    infile lsinfo pad end=last;

    input rec $250.;

    dsn=scan(rec,-1,' ');

    owner=scan(rec,3,' ');

    permissions=scan(rec,1,' ');

  size=scan(rec, 5, ' ');

run;

[sasprofile]

Reeza thanks for the valuable information

am completely new to the Unix Solaris, so am not sure how to create Daily report.

so the same code you provided need to run as .sas program on solaris server to get the daily reports

[jakarman]

reeza, your are right.   Normally you can have logging like nmon/topas being active making snapshots. Mostly every 5 minutes so you can see afterwards what is happening.  What sasprofle is saying all mountpoints are filled up at the same  moment. The mountpoints/filesystems are visible with the "df -g" command.

saswork and several others should be isolated. It doesn't make sense when all of them get filled up on the same moment.

Having a 200Gb or 500Gb of saswork you can easily seen that growing/filling up as IO speed is not that fast it is capable to do that very quickly it will take as an hour.

The last part of filling up for 80 to 100% will slow down dramatically as the filesystem is needing harder work to find free space.

If the behavior is different something else could be causing that.

Corruptions in the SAN an filesystem could happen and cause this kind of OS system failures. Yup I have experienced  those on this kind of machines. All hardware and OS have their failures and at some level lack of reliability.  One of those was a lost disk - unknown reason another was failure in DAC rights processing, never found the cause.   

[jakarman]

saspronfile, "main mounting point /sas" . "others like /sas/work below" ??   Unix is different to Windows in storage management. This makes no sense.

With Unix you have storage and you can map that to a directory/folder that is a mountpoint.. With Windows you have a drive/storage where you can define directories folders. With a DFS you can present many of those as as single directory structure. That is a bit different way of thinking.

With a HFS you have those mounpoints and the security inheritance by all level. this is why there is a standardized structure like General overview of the Linux file system

- /opt for third party software eg SAS  (with rpm also bin)

- /var for temporary data eg    saswork and logging.

- /home for personal "my documents"

- ....

SAS institiute did made a complete mess of this common approach and guidelines by ignoring those. This is commonly causing a lot of trouble needing cooperation by unix admins. You are running out of space/resources. The firtst one is what are the mountpoints.... "df -g" and how big it he work/util?

[sasprofile]

Please can any one help me with the below requirement for Solaris 5.10 machine

and I want daily report on files in the space sorted by size and then one sorted by last date accessed?
actually I want to create a space management report in SAS. How can we do that ?

I would really appreciate all your help

[Reeza]

Did you try the code I provided?

[sasprofile]

Reeza

This is the code you provided,so do you want me to run this code on my Solaris machine as .sas program by saving it with .sas program.

please help me with this

Use Unix commands to read the directory information;

filename lsinfo pipe 'cd /folder1/folder2; ls -l';

data work_space;

    infile lsinfo pad end=last;

    input rec $250.;

    dsn=scan(rec,-1,' ');

    owner=scan(rec,3,' ');

    permissions=scan(rec,1,' ');

  size=scan(rec, 5, ' ');

run;

[Reeza]

Yes, change the folder to the path to the work folder and see the output. Are you the admin on the box? If not, perhaps talk to the admin who should understand the basic commands to generate a file listing of a folder.

Either create a SAS or batch script that gets the file listings in the folder.

Schedule that as a job to run every hour or half hour for a week and see what happens.

Given all the possible givens for a set up it's hard to help beyond that.

[Kurt_Bremser]

This code:

%let basepath=/whatever/subdir;

filename oscmd pipe "find &basepath -xdev -type f -exec ls -ul {} \;";

%let actyear = %sysfunc(year(%sysfunc(date())));

data files;

infile oscmd truncover;

length

  perms $ 10

  links 3

  user $ 8

  group $ 8

  size 8

  month $ 3

  day $ 2

  yrtm $ 5

  full_path $ 200

  fname $ 100

  path $ 200

  acc_date 5

;

input

  perms

  links

  user

  group

  size

  month

  day

  yrtm

  full_path

;

format acc_date date9.;

fname = scan(full_path,-1,'/');

index = findc(full_path,'/','b');

if index > 0 then path = substr(full_path,1,index - 1);

if length(yrtm) = 5

then acc_date = input(day !! month !! "&actyear", date9.);

else acc_date = input(day !! month !! substr(yrtm,1,4),date9.);

drop yrtm day month full_path index;

run;

will provide you with a dataset that contains (among other values) the filename, the path to the file, the file size, and the last access date. For everything in a given directory tree and the same physical filesystem.

[sasprofile]

KurtBremser,

Thank you so much for all your help,I truly appreciate that

The data set got created  with all the information like perms links user group size fname path and acc_date when I ran the code from EG.

when I ran the code with basepath= /sas/saswork I have seen files with my user id access_date Aug 27 in the dataset output,but I don't see any files with the name of the other couple of users and a sas id which actually got created when I checked /sas/saswork on my solaris server.

and also when I put the basepath=/sas and ran the code which is the main mounting point in  which all other mounting points comes,but when am checking the dataset I can see files only with access date of Aug 23rd and I don't see anything after that.

atleast it should show the /saswork file accessed with todays date right,so am not sure whether am doing something wrong.

and also one more request is there anyway that this dataset can give output with the file size in MB or GB,can i schedule the same code on my Solaris server for every 1/2 hour and is there anyway it can produce the output in graphical way or any other way instead of dataset.

Thanks you so much for all your help again

[Reeza]

Are all files visible in /sas/saswork when you ls or view the directory in the data set?

Size is in bytes, simple division will convert to MB and GB.

The work library resets whenever a SAS session is closed so if someone isn't currently using SAS they shouldn't have any files in the WORK library.