topic Re: Help with tunning - sas disk space in SAS Procedures

Help with tunning - sas disk space

Augusto — Wed, 12 Aug 2015 02:25:33 GMT

Hello guys,

I need a big help.

Here at work, we identified some ways to optimize our disk space.

Below I have listed what has been identified as well and what needs to be done. I would like to know of you how to do it

a) Identifield: Sas data file without access by users

b) Needed: List identifying the sas data file with no access..

a) Identifield: Filesystems unused

b) Needed: List with identification of unused Filesystems.

a) Identifield: Existence of compacted sas data file

b) Needed:List with identification of compacted sas data files and processes that use them.

a) Identifield: Existence of sas data file with the same name

b) Needed: List identifying the sas data files with the same name but in different directories

a) Identifield: Compression sas data file

b) Needed: A list of sas data file which may be compressed

a) Identifield: Shared filesystems

b) Needed: A list of Shared filesystems

If you have some suggestions for best practices, please feel free to share,

Thanks in advance.

Re: Help with tunning - sas disk space

jakarman — Wed, 12 Aug 2015 05:21:34 GMT

Augusto, your list is looking like some chosen checkpoint made by an auditor.

1/ data without access by users

*) Needed: List identifying the sas data file with no access..

This is monitoring of all activity on all data on the OS level. With Arm log4j being active SAS can do a lot of this.

You only can get a list if data that is used. Unused ones will cause no events. Use the list of all data that is present with that. (3,4,5)

2) Unused space / Filesystems

*) Needed: List with identification of unused Filesystems.

This is an OS task (xcmd usage) the df command woll show all filesystems. With a decent setup you should see what is stores where and how much is free.

The du command will make a list of the space in use.

3) Existence of compacted sas data file

*) Needed:List with identification of compacted sas data files and processes that use them.

Make a list of all directories with sas-datasets. The attributes off all datasets can be gathered as a new datasets. One of the elements is compress

http://support.sas.com/documentation/cdl/en/proc/67916/HTML/default/viewer.htm#p1sy9ca8n2tv03n1savk4p9cmeq0.htm#p17vojw133aeyfn1u8modmp2mig9

Also available as a SASHELP.v----- view member.

The files that are compacted using zip or another to be analyzed at OS elvel. Make a list at OS level (ls command using xmcd) of all OS files (recursive option is there)

4) Existence of sas data file with the same name

* ) Needed: List identifying the sas data files with the same name but in different directories

Having a list of all OS files have that on split in path name and filename. Just ordering will give you an answer

5) Compression sas data file

*) Needed: A list of sas data file which may be compressed

Having a list of all SAS datasets with all their attributes. look for the ones that are not compressed and having long records/many variables many observations

6) Shared filesystems

*b) Needed: A list of Shared filesystems

Shared filesystems is an OS task. At Windows it is very common usage to propagate to a desktop. Within in a –Nix system it is a “no done” situation as of too many possible security holes. (mostly forbidden)

With a SAS grid approach it could be there. Check with the storage admin / system admin.

Re: Help with tunning - sas disk space

Kurt_Bremser — Wed, 12 Aug 2015 07:07:29 GMT

4) (UNIX)

filename oscmd pipe 'find / -name \*.sas7bdat -print';

data files;

infile oscmd lrecl=500 truncover;

length

path $500

fname $200

;

input path;

fname = lowcase(scan(path,-1,'/'));

run;

proc sort data=files;

by fname;

run;

data int (keep=fname);

set files;

retain count;

if first.fname then count=1; else count+1;

if last.fname and count > 1 then output;

run;

data result;

merge

files

int (in=_int)

;

by fname;

if _int;

run;

Get the attributes from sashelp.vtables (make sure that ALL directories containing SAS fileshave a libname assigned, as sashelp.vtables is dynamically created from all currently defined&assigned libraries.

Re: Help with tunning - sas disk space

Kurt_Bremser — Wed, 12 Aug 2015 07:11:34 GMT

BTW: please refrain from double-posting. Makes only life harder for all involved.

Re: Help with tunning - sas disk space

LinusH — Wed, 12 Aug 2015 08:05:00 GMT

I think that Jaap covered the bullets quite extensively.

A general thought. It sounds like that your users have quite some freedom to create data. Instead of chasing ghosts you could centralise your data management. Allowing data to be stored in specific locations.