How to get the file size of a spde file on a Unix server

[alepage]

Good Morning,

I have test this macro function that should give the file zise but it is not working.

Please note that we have spde files partitions like 

/.../auto/be_auto_prmsep2024.dpf.00010a84.0.2.spds9
/.../auto/be_auto_prmsep2024.dpf.00010a84.1.2.spds9
/.../auto/be_auto_prmsep2024.dpf.00010a84.2.2.spds9

But I would like to get the file size of be_auto_prmsep2024 not of each partition.

I have tried the following SAS function and it is not working

%macro FileAttribs(filename);           
  %local rc fid fidc Bytes CreateDT ModifyDT; 
  %let rc=%sysfunc(filename(onefile,&filename)); 
  %let fid=%sysfunc(fopen(&onefile));
     %if &fid ne 0 %then %do;  
  %let Bytes=%sysfunc(finfo(&fid,File Size (bytes)));   
  %let CreateDT=%sysfunc(finfo(&fid,Create Time));     
  %let ModifyDT=%sysfunc(finfo(&fid,Last Modified));    
  %let fidc=%sysfunc(fclose(&fid));    
  %let rc=%sysfunc(filename(onefile)); 
  %put NOTE: File size of &filename is &bytes bytes;    
  %put NOTE- Created &createdt;       
  %put NOTE- Last modified &modifydt;
     %end;
        %else %put &filename could not be open.;
%mend FileAttribs;
                      
%FileAttribs(c:\aaa.txt)

I would like to create a table like below

path                                        filename                             size

path1                                     be_auto_prmsep2024     555 kbytes

path1                                     be_auto_prmoct2024     545 kbytes

and so on.

how do we do that ?

[ChrisNZ]

If all files are in the same directory, list all the files in that directory (see the pinned topic https://communities.sas.com/t5/SAS-Programming/How-to-List-all-the-files-in-a-folder/m-p/674065#M202...) and get the size of all component files. Don't forget index files if applicable.

Note that you can use the PARTSIZE option of LIBNAME SPDE to avoid splitting the data into multiple files.

[Kurt_Bremser]

The SPDE engine is meant to speed up processing by spreading it over multiple storage devices, so you may find that your dataset resides in multiple directories.

So your first task would be to find all the partition files, after which you can determine each file's size and add that up.

Switch to doing it all in a DATA step, where you can use all the functions without a "%SYSFUNC avalanche".

[ChrisNZ]

@Kurt SPDE is also great to speed up IO thanks to its much better compression algorithms and better indexing, even if just one directory is used.

[alepage]

Good Morning Mr. Bremser,

Usually, the premium datasets for a specificy year, company (ex: ha for Halifax), lob (auto, prop, entr) reside into the same folder.

ex: /.../sasdata/sas2024/ha/auto Contains all the premium datasets for auto, year=2024, months=all and lob=auto.

ha_auto_prmjan2024

ha_auto_prmfev2024

...

ha_auto_prmdec2024

As we use the spde file engine, these files are partionned but in the same folder as described above.

So what I need to do, it is to create only on SAS dataset that will give me the following information:

path                         fname               sum of the size for all the partions (ex: sum of all the partitions for ha_auto_prmjan2024)

and that for all year ( 2001 - 2024) , all cies, lob=auto, prop, entr, 

That dataset will give me the space used from the actual datasets as well as an estimation of the space required to copy those into another folder for a specific project and a limited period of time. How do we do that? Please note that we have unix serser, so many script found on the SAS community are useless.

[SASJedi]

I have a macro I use for this. My personal macro library has lots of utility macros in it, and some macros use other macros in the set. The code below will compile the required macros straight from the GitHub repository.

CAVEAT: You should preview the contents of each these files at https://github.com/SASJedi/sas-macros before using them this way so that you can be confident that the code you are executing is safe for your intended use.  

filename code url "https://raw.githubusercontent.com/SASJedi/sas-macros/refs/heads/master/exist.sas";
%include code/source2;
filename code url "https://raw.githubusercontent.com/SASJedi/sas-macros/refs/heads/master/fileattribs.sas";
%include code/source2;
filename code url "https://raw.githubusercontent.com/SASJedi/sas-macros/refs/heads/master/findfiles.sas";
%include code/source2;
filename code url "https://raw.githubusercontent.com/SASJedi/sas-macros/refs/heads/master/spdefilesize.sas";
%include code/source2;
filename code url "https://raw.githubusercontent.com/SASJedi/sas-macros/refs/heads/master/translate.sas";
%include code/source2;

%spdeFileSize(?)

This call to the macro:
 %spdeFileSize(?) 
will write syntax help to the SAS log, including a usage example:

 NOTE: SPDEFILESIZE documentation:
      
       Purpose: For an SPDE library dataset, find the filesize for each
                individual sub-file, and the sum of the individual sizes.
      
       Syntax: %SPDEFILESIZE(libref,dsn)
       libref: Libref of the SPDE library
       dsn:    Name of the dataset in the SPDE library
      
       Example: %SPDEFILESIZE(spdelib,myTable)
       Result:
      
       |-------------------------------------------------------------|
       |File                                       |File size (bytes)|
       |-------------------------------------------------------------|
       |/home/myID/spde1/mytable.dpf.a86.2.22.spds9|       98,572,032|
       |/home/myID/spde2/mytable.dpf.a86.0.22.spds9|      134,217,216|
       |/home/myID/spde3/mytable.dpf.a86.1.22.spds9|      134,217,216|
       |Total for all segments:                    |      367,006,464|
       |-------------------------------------------------------------|
      
       Use ? to print this documentation to the SAS log.

[yabwon]

Hey Mark (@SASJedi),

Off topic thing.

How about building a package out of your SAS-Jedi macros library?

Bart

[SASJedi]

@yabwon  I'm looking into that. It would make sharing a lot easier, because my macros often call one or more of the others as modular code. 

[yabwon]

Maybe the "Swiss army knife" a.k.a. Proc Datasets?

libname a SPDE "R:\";

data a.x a.y a.z;
  array m[20] (1:20);
  do i=1 to 2e6;
    output;
  end;
run;

/*
proc contents data=a.x DETAILS;
run;
*/
ods trace on;
ods select all; 
ods output Members=Members; /* data set */
proc datasets lib=a;
run;
quit;
ods trace off;

[yabwon]

One more option: from the BasePlus SAS Package macro %sdirsAndFiles(),

Something like this:

libname a SPDE "R:\test\";

data a.x(index=(i)) a.y a.z(index=(im=(i m1)));
  array m[20] (1:20);
  do i=1 to 2e6;
    output;
  end;
  format i dollar10.2;
run;


filename packages "c:\sas_work\sas_packages";
%include packages (SPFinit.sas);
%loadPackage(BasePlus)

%dirsAndFiles(
    R:\test\
   ,ODS=x
   ,details=1
   ,keepDirs=0
   ,keepFiles=1
   ,maxDepth=1
)
proc print data=x;
run;

Gives:

Bart

[Ksharp]

You need to use the UNIX command 'du' to get these file size.

I wrote a article about it before. Here is:

Did you know how big your library is ?



 

 

During our data maintenance , we usually want to know how big our libraries are. Which libraries dominate the whole disk.

 

First of all, I would like to introduce the SAS environment we need to processs. Our SAS9.2 SPD Server is under AIX operation system and we use EG4.3 to connect to SAS server by using Integrated Open Model (IOM).

 

I know there is a command SPDSLS to get the library’s size , but it only handle one library one time. If we has more than one hundred libraries to check , this way is too hard for us.  I found UNIX command DU is very convenient and flexiable , I think we can use it to do better job.

 

A SPD Server library contain two part : one is Meta data which is really small ,we can neglect it ; the other is real table data which we need to check.

 

Now the following is the step by step how we complete this task, Hope can help other peoples.

 

1) Running the following code in EG.

 

options nolabel;

proc sql;

create table x as

 select libname,engine,sysname,'du -g '||strip(sysvalue)||' > '||strip(libname)||'.txt' as value length=400 

  from dictionary.libnames

   where sysname='Datapath' and engine='SASSPDS'

order by libname;quit;

 

 

 

We can see this table now.

 

 

 

 

The SQL create a table named x which contain name of library, engine of library , the type of parameter and a command you used to calculated the size of library.

From sysname=”Datapath”, we know sysvalue is the data path of storing for this libname.

 

Value column really look like :

du -g '/spds_data1/bhyh1/' '/spds_data2/bhyh1/' '/spds_data3/bhyh1/' '/spds_data4/bhyh1/' '/spds_data5/bhyh1/' '/spds_data6/bhyh1/' '/spds_data7/bhyh1/' > BHYH1.txt 

 

“du –g” means calculated the directory’s size by G measure (e.g. 100 G ).  The middle is the data path (i.e. directory ).  “  > BHYH1.txt  ” is telling OS to redirect the result of the command into a txt file.

 

2) At AIX , create a temporary directory :

 

Using software secureCRT to connect to SAS SPD Server . and create a lib_size directory under /spds_data11/temp/

 

 

3)Enter the directory we create a moment ago .

 

 

 

4)Copy these command from dataset X .

 

 

 

 

5) Paste these command into secureCRT  and enter ENTER :

 

 

After that you will see lots of txt file under directory lib_size.

 

 

Open one of them , find the txt look like :

 

 

6) Running the following code to get  the finally result we need , easy Hoo ?

 

data temp;

 infile '/spds_data11/temp/lib_size/*.txt' lrecl=400 expandtabs ;

 input size path : $40. ;

 length lib_name  $ 40 ;

 lib_name=upcase(scan(path,-2,'/'));

run;

proc means data=temp nway noprint;

class lib_name;

var size;

output out=want(drop=_:) sum=lib_size;

run;

 

 

Here is the final result.