How to keep track of space used and free space during the job execution

[odmhx]

I have a long and big production job run on the weekends. The space used at the peak (sort and merge) varies from 200 gb to 240 gb. However when it's close to the year end, as the input files become bigger and bigger, causing the peak use of space exceeding the 250 gb limit thus abend.

Can anyone give me some suggestion/coding to monitor the peak space usage of the program, i.e, write it to the log so I can pro-actively monitor the space issue and request more space before it reaches the limit?

I am running this job in the background on an AIX server.

Thank you in advance

[DanielSantos]

Hi.

There is more than a way of doing this, but I think, any of them will rely on reporting of your OS.

The following AIX command will report total, available and used storage under a specified /dir

[pre]df -g /dir[/pre]
the -g modifier is for GB unit

I guess you could pipe the result into SAS and output it to the log, something very simple like this:

[pre]%macro df(DIR);
data _null_;
infile 'df -g data' pipe;
input;
put _infile_;
run;
%mend;[/pre]
Then you just have to call the macro, for example, before/after every datasetp/proc submited in your program, like this:

[pre]proc sort data=...
...
run;

%df(/home/c085442/data);

data ...
...
run;

%df(/home/c085442/data);[/pre]
Another way would be to write a shell script that using a specified time interval would issue the command and then write the result (timestamped) into a file.
From there you could issue some kind of warning or even send a mail to a list of specified recipients.

Hope it helps.

Cheers from Portugal

Daniel Santos @ www.cgd.pt

[odmhx]

Thank you very much for your reply. I also thought of your first solution, but didn't use it because it doesn't run in the middle of the data step or proc sort where the abend happens. When the input file size increases, it seems that the working storage expands exponentially in the merge and sort steps, that makes gestimate of the space usage very hard.

Your second suggestion is very helpful. I'll try that.


> Hi.
>
> There is more than a way of doing this, but I think,
> any of them will rely on reporting of your OS.
>
> The following AIX command will report total,
> available and used storage under a specified /dir
>
> [pre]df -g /dir[/pre]
> the -g modifier is for GB unit
>
> I guess you could pipe the result into SAS and output
> it to the log, something very simple like this:
>
> [pre]%macro df(DIR);
> data _null_;
> infile 'df -g data' pipe;
> input;
> put _infile_;
> run;
> %mend;[/pre]
> Then you just have to call the macro, for example,
> before/after every datasetp/proc submited in your
> program, like this:
>
> [pre]proc sort data=...
> ...
> run;
>
> %df(/home/c085442/data);
>
> data ...
> ...
> run;
>
> %df(/home/c085442/data);[/pre]
> Another way would be to write a shell script that
> using a specified time interval would issue the
> command and then write the result (timestamped) into
> a file.
> From there you could issue some kind of warning or
> even send a mail to a list of specified recipients.
>
> Hope it helps.
>
> Cheers from Portugal
>
> Daniel Santos @ www.cgd.pt

[sbb]

Investigate using the SAS-maintained view identified as SASHELP.VTABLE, where you will find useful statistics on this type of information. Search the SAS support website and do a PROC CONTENTS on the view for details.

Scott Barry
SBBWorks, Inc.

[odmhx]

Thank you. I will.

> Investigate using the SAS-maintained view identified
> as SASHELP.VTABLE, where you will find useful
> statistics on this type of information. Search the
> SAS support website and do a PROC CONTENTS on the
> view for details.
>
> Scott Barry
> SBBWorks, Inc.