GZIP all SAS datasets in a library.

[adamhyman]

I'd like to use SAS to Zip all the datasets in a particular library.

Running   systask command "gzip '/sas/my special library/mydataset.sas7bdat'" wait taskname=zip status=check shell;   does work correctly!!

But I'm struggling to write the code to get it to iterate over each dataset.

This has been my noble, but so-far-unsuccessful attempt:

I have the following code which puts the names of datasets into a macro array.

data datasets;

  set sashelp.vmember (where=(libname='my_lib' and memtype='DATA'));

  run;

%array(datasets, data=datasets, var=memname);

Now I would like to repeat this code...

systask command "gzip '/sas/my special library/mydataset.sas7bdat'" wait taskname=zip status=check shell;

... using the values in the macro array instead of "mydataset"

I tried this, which doesn't work, because the ? is between single quotes, so isn't converted to the macro variables:

%do_over(files, phrase= systask command "gzip '/sas/cc_data/adamh/20131014 - Unidentified Callers/?.sas7bdat'"

                        wait taskname=zip status=check shell; );

Any advice would be greatly appreciated!!

[Reeza]

Create a macro with the gzip commands instead and then call in a data step with call execute?

Untested below....

%macro gzip(file_name);

systask command "gzip '/sas/my special library/&file_name..sas7bdat'" wait taskname=zip status=check shell;

%mend;

data _null_;

set datasets;

call execute("%gzip("||memname||");");

run;

[ChrisHemedinger]

Here's a way to accomplish it without the need for GZIP.  This uses ODS PACKAGE (available in SAS 9.2 and later) to create the zip file for you.  The first part of the example "sets up" a small library/folder with just a couple of data members.  It uses this LIBNAME trick to create a new folder.

The last part uses FILENAME ZIP to modify the zip file a bit to remove a PackageMetaData artifact -- that's optional.  You need SAS 9.4 to use FILENAME ZIP.

/* This sets up a folder that contains just the data we want */

options dlcreatedir;

libname tozip "%sysfunc(getoption(work))/zip";

proc datasets lib=tozip;

copy in=sashelp out=tozip;

select cars class;

quit;

/* End of setup */

/* generate series of ODS PACKAGE ADD statements */

/* One for each data set file                    */

proc sql;

select

   catt('ods package(datazip) add file="',path,'/',lowcase(memname),'.sas7bdat";')

  into: datafiles separated by ' '

  from sashelp.vmember

  where libname='TOZIP' and memtype='DATA';

quit;

/* Creating a ZIP file with ODS PACKAGE */

ods package(datazip) open;

&datafiles;

ods package(datazip) publish archive

  properties(

   archive_name="tozip.zip"

   archive_path="c:\temp"

  );

ods package(datazip) close;

/* If you have SAS 9.4, you can use FILENAME ZIP           */

/* To remove the "PackageMetaData" item from your ZIP file */

filename pkg ZIP "c:\temp\tozip.zip" member="PackageMetaData";

data _null_;

  if (fexist('pkg')) then

  rc = fdelete('pkg');

run;

filename pkg clear;

Chris

[Kumarz]

Chris,

Is FILENAME GZIP supported in SAS 9.4?

I mean, can I directly use FILENAME GZIP to zip SAS/other files?

Kumar.

[ChrisHemedinger]

FILENAME ZIP is what's supported, which achieves what you want, I think.

Reading and updating ZIP files with FILENAME ZIP - The SAS Dummy

Chris

[Kumarz]

Chris,

Thanks for the reply.

I should have asked this : Do we need to have ALLOW XCMD option configured in our SAS environment to support FILENAME ZIP?

As of now, we are not allowing our users to run system commands; so with 9.4 can user use FILENAME ZIP even with DISABLED ALLOW XCMD?

Kumar.

[jwillis]

Chris,

How do I read the SAS datasets stored in tozip.zip?

[ChrisHemedinger]

James,

This blog post shows how to use FILENAME ZIP to extract content from a ZIP file:

http://blogs.sas.com/content/sasdummy/2014/01/29/using-filename-zip/

There are some comments in the post that have some helpful variations.  There isn't an "Unzip" function in SAS, so pulling data sets from the ZIP file will involve some brute-force copy methods that you might need to play around with.  See some examples in this article.

Chris

[Vish33]

Hi,

Try this code. with out any macros...you can run this single statement to gzip all datasets present in that particular library.

X gzip "/sas/my special library/*.sas7bdat";

[Kurt_Bremser]

This will fail if there are too many datasets in the library (command line limitations).

[Vish33]

Hi,

May i know what is the limit for number of datasets.

Regards,

Vish

[Kurt_Bremser]

As Jaap said, the limit is the size of the command line buffer, which is usually 4K.

Since the shell uses this buffer to expand the wildcards, gzip *.sas7bdat may lead to "parameter list too long".

My version hands the * over to the find command, which does not expand the list, but instead matches the pattern against any file it finds. The files are then gzip'd one by one, a method that scales indefinitely, for all purposes.

If you have to operate under a noxcmd restriction, you may still be able to ssh to the server and submit the command there.

If I can solve a problem with a single shell command, I prefer that every time to a lengthy SAS code.

Many problems can be solved with a hammer, but sometimes the screwdriver is better.

[Kurt_Bremser]

Why not do it completely in the shell?

x 'find "/sas/my special library" -maxdepth 1 -name \*.sas7bdat -exec gzip {} \;';

[jakarman]

@Kumarz  Just a question: Why are you not allowed to use xcmd?

a/ Because of the default setting in the SAS installation and your support/installation staff do not want to think about your requirements to get your work done? Than you have a hayman reasoning. 

b/ Because your are not allowed to do that kind of work with that data.  So, you have no problem as you are supposed not to do that.

As there are a lot of functions in SAS to access data of any type that pose an other question.

Did your organization anything about data-classification data-policies and something like RBAC (Role Based Accesc Control)?

@Vish33, There are limits at Unix commands as they are often using a fixed 4k internal buffer. The way KurtBremser solved I expect there are no limitations on that side. Zip formats also are having their limits on number of datasets and size of datasets. Not all zip tools are equal. http://en.wikipedia.org/wiki/Zip_(file_format)

[Kumarz]

Jaap,

I would say :

a/ --> Because of the default setting in the SAS installation and our support/installation staff do not want us to run system commands as they are worried about users bringing down the server or creating issues by running system commands accidentally (of course we don't have access/permissions to change resources on the server). But they are saying that allowing users to run System Commands is not a standard practice in SAS and doing that we are opening a new risk window by allowing users to run NON-REQUIRED system commands from SAS.

b/ --> I can go to the CLI and run ZIP/GZIP command on the server to compress my data and I'm allowed to do that. But I'm unable to do that if I try to use X command from SAS EG.

Since we are kind of a new SAS site; as of now we don't have data-classification or data-policies and something like RBAC (Role Based Accesc Control). I would appreciate if you suggest something here which I can take to my SAS staff and management to make sure we get what we want without creating unnecessary risks/issues for our IT.

Thanks,

Kumar.