topic Re: Measure disk usage in SAS Procedures

Measure disk usage

Florent — Wed, 10 Nov 2010 10:25:02 GMT

All,

I would like to know whether it exists a mean to measure the maximum (peak) usage of the disk space used by my work library during the execution of one of my programs (knowing that in this program we try to delete the maximum of datasets during execution in order to free the maximum disk space as possible).

The aim is to predict the possible lack of space in case of big data processing.

Note: I'm using a remote connection to a windows NT server with SAS 9.1. The pc used to remotely connect to this server has SAS 8.

Thanks for your help.

Kind regards,
Florent

Re: Measure disk usage

Peter_C — Wed, 10 Nov 2010 17:58:24 GMT

might be easier if you have rights to invoke perfmon on the server, or know an administrator who has.

Of course between steps you can get the total space used by objects in your work library
proc datasets details; run;
reveals the data, but I find no sashelp.vxxxxxx providing the info so you would need to trap the table through ODS OUTPUT

However, it won't reveal the peak usage that happens during a step - e.g. when
proc sql performs a cartesian join before grouping results
or proc means needs to summarise on disk

Re: Measure disk usage

andreas_lds — Thu, 11 Nov 2010 08:45:05 GMT

Maybe i am missing something, but sashelp.vtable seems to contain the required data:
[pre]
proc sql print;
select sum(filesize) as SpaceUsed
from sashelp.vtable
where Libname = 'WORK' and MemType = 'DATA'
;
quit;[/pre] pre-tags added

Message was edited by: andreas_lds

Re: Measure disk usage

Peter_C — Thu, 11 Nov 2010 12:20:15 GMT

Andreas
sashep.vtable doesn't reveal the sizes for members of the library of types other than tables :
work is used not only for tables but also for catalogs, and significantly in SAS9, item stores e.g. for ODS. Also stored in work are utility files created when running steps like proc sort. Other utility files created in work hold content of display manager list and log screens. There are many other objects of types relevant to SAS solutions of which I'm aware but know little. That is why I didn't think sashelp.vtable enough to report the maximum disk-space usage of a library.
Even the details report from proc datasets might not be enough as it does not report the utility files.
Reporting sizes even from proc datasets would not catch the balooning usage of space in the work library during a proc sql step that requires an unrestricted cartesian join - space that might be released before the sql processing completes.

Hence my proposal of perfmon monitoring the server.
peter

Re: Measure disk usage

Florent — Mon, 15 Nov 2010 16:13:48 GMT

Thanks to both of you for the feedback !

Regards,
Florent

Re: Measure disk usage

CameronLawson — Thu, 25 Nov 2010 11:38:42 GMT

It also depends on the proc statement / datastep used. For example; a proc sort in some cases requires at least 3 times the dataset size in space.