topic Re: GZIP all SAS datasets in a library. in SAS Programming

GZIP all SAS datasets in a library.

adamhyman — Thu, 17 Oct 2013 23:39:51 GMT

I'd like to use SAS to Zip all the datasets in a particular library.

Running systask command "gzip '/sas/my special library/mydataset.sas7bdat'" wait taskname=zip status=check shell; does work correctly!!

But I'm struggling to write the code to get it to iterate over each dataset.

This has been my noble, but so-far-unsuccessful attempt:

I have the following code which puts the names of datasets into a macro array.

data datasets;

set sashelp.vmember (where=(libname='my_lib' and memtype='DATA'));

run;

%array(datasets, data=datasets, var=memname);

Now I would like to repeat this code...

systask command "gzip '/sas/my special library/mydataset.sas7bdat'" wait taskname=zip status=check shell;

... using the values in the macro array instead of "mydataset"

I tried this, which doesn't work, because the ? is between single quotes, so isn't converted to the macro variables:

%do_over(files, phrase= systask command "gzip '/sas/cc_data/adamh/20131014 - Unidentified Callers/?.sas7bdat'"

wait taskname=zip status=check shell; );

Any advice would be greatly appreciated!!

Re: GZIP all SAS datasets in a library.

Reeza — Fri, 18 Oct 2013 02:01:33 GMT

Create a macro with the gzip commands instead and then call in a data step with call execute?

Untested below....

%macro gzip(file_name);

systask command "gzip '/sas/my special library/&file_name..sas7bdat'" wait taskname=zip status=check shell;

%mend;

data _null_;

set datasets;

call execute("%gzip("||memname||");");

run;

Re: GZIP all SAS datasets in a library.

ChrisHemedinger — Fri, 18 Oct 2013 12:55:56 GMT

Here's a way to accomplish it without the need for GZIP. This uses ODS PACKAGE (available in SAS 9.2 and later) to create the zip file for you. The first part of the example "sets up" a small library/folder with just a couple of data members. It uses this LIBNAME trick to create a new folder.

The last part uses FILENAME ZIP to modify the zip file a bit to remove a PackageMetaData artifact -- that's optional. You need SAS 9.4 to use FILENAME ZIP.

/* This sets up a folder that contains just the data we want */

options dlcreatedir;

libname tozip "%sysfunc(getoption(work))/zip";

proc datasets lib=tozip;

copy in=sashelp out=tozip;

select cars class;

quit;

/* End of setup */

/* generate series of ODS PACKAGE ADD statements */

/* One for each data set file */

proc sql;

select

catt('ods package(datazip) add file="',path,'/',lowcase(memname),'.sas7bdat";')

into: datafiles separated by ' '

from sashelp.vmember

where libname='TOZIP' and memtype='DATA';

quit;

/* Creating a ZIP file with ODS PACKAGE */

ods package(datazip) open;

&datafiles;

ods package(datazip) publish archive

properties(

archive_name="tozip.zip"

archive_path="c:\temp"

);

ods package(datazip) close;

/* If you have SAS 9.4, you can use FILENAME ZIP */

/* To remove the "PackageMetaData" item from your ZIP file */

filename pkg ZIP "c:\temp\tozip.zip" member="PackageMetaData";

data _null_;

if (fexist('pkg')) then

rc = fdelete('pkg');

run;

filename pkg clear;

Chris

Re: GZIP all SAS datasets in a library.

Kumarz — Mon, 28 Apr 2014 20:32:56 GMT

Chris,

Is FILENAME GZIP supported in SAS 9.4?

I mean, can I directly use FILENAME GZIP to zip SAS/other files?

Kumar.

Re: GZIP all SAS datasets in a library.

ChrisHemedinger — Tue, 29 Apr 2014 01:37:31 GMT

FILENAME ZIP is what's supported, which achieves what you want, I think.

Reading and updating ZIP files with FILENAME ZIP - The SAS Dummy

Chris

Re: GZIP all SAS datasets in a library.

Vish33 — Tue, 29 Apr 2014 09:39:33 GMT

Hi,

Try this code. with out any macros...you can run this single statement to gzip all datasets present in that particular library.

X gzip "/sas/my special library/*.sas7bdat";

Re: GZIP all SAS datasets in a library.

Kurt_Bremser — Tue, 29 Apr 2014 10:34:53 GMT

This will fail if there are too many datasets in the library (command line limitations).

Re: GZIP all SAS datasets in a library.

Kurt_Bremser — Tue, 29 Apr 2014 10:38:51 GMT

Why not do it completely in the shell?

x 'find "/sas/my special library" -maxdepth 1 -name \*.sas7bdat -exec gzip {} \;';

Re: GZIP all SAS datasets in a library.

Kumarz — Tue, 29 Apr 2014 14:15:14 GMT

Chris,

Thanks for the reply.

I should have asked this : Do we need to have ALLOW XCMD option configured in our SAS environment to support FILENAME ZIP?

As of now, we are not allowing our users to run system commands; so with 9.4 can user use FILENAME ZIP even with DISABLED ALLOW XCMD?

Kumar.

Re: GZIP all SAS datasets in a library.

Vish33 — Tue, 29 Apr 2014 15:13:05 GMT

Hi,

May i know what is the limit for number of datasets.

Regards,

Vish

Re: GZIP all SAS datasets in a library.

jakarman — Tue, 29 Apr 2014 17:30:28 GMT

@Kumarz Just a question: Why are you not allowed to use xcmd?

a/ Because of the default setting in the SAS installation and your support/installation staff do not want to think about your requirements to get your work done? Than you have a hayman reasoning.

b/ Because your are not allowed to do that kind of work with that data. So, you have no problem as you are supposed not to do that.

As there are a lot of functions in SAS to access data of any type that pose an other question.

Did your organization anything about data-classification data-policies and something like RBAC (Role Based Accesc Control)?

@Vish33, There are limits at Unix commands as they are often using a fixed 4k internal buffer. The way KurtBremser solved I expect there are no limitations on that side. Zip formats also are having their limits on number of datasets and size of datasets. Not all zip tools are equal. http://en.wikipedia.org/wiki/Zip_(file_format)

Re: GZIP all SAS datasets in a library.

Kurt_Bremser — Tue, 29 Apr 2014 19:21:31 GMT

As Jaap said, the limit is the size of the command line buffer, which is usually 4K.

Since the shell uses this buffer to expand the wildcards, gzip *.sas7bdat may lead to "parameter list too long".

My version hands the * over to the find command, which does not expand the list, but instead matches the pattern against any file it finds. The files are then gzip'd one by one, a method that scales indefinitely, for all purposes.

If you have to operate under a noxcmd restriction, you may still be able to ssh to the server and submit the command there.

If I can solve a problem with a single shell command, I prefer that every time to a lengthy SAS code.

Many problems can be solved with a hammer, but sometimes the screwdriver is better.

Re: GZIP all SAS datasets in a library.

Kumarz — Tue, 29 Apr 2014 20:59:07 GMT

Jaap,

I would say :

a/ --> Because of the default setting in the SAS installation and our support/installation staff do not want us to run system commands as they are worried about users bringing down the server or creating issues by running system commands accidentally (of course we don't have access/permissions to change resources on the server). But they are saying that allowing users to run System Commands is not a standard practice in SAS and doing that we are opening a new risk window by allowing users to run NON-REQUIRED system commands from SAS.

b/ --> I can go to the CLI and run ZIP/GZIP command on the server to compress my data and I'm allowed to do that. But I'm unable to do that if I try to use X command from SAS EG.

Since we are kind of a new SAS site; as of now we don't have data-classification or data-policies and something like RBAC (Role Based Accesc Control). I would appreciate if you suggest something here which I can take to my SAS staff and management to make sure we get what we want without creating unnecessary risks/issues for our IT.

Thanks,

Kumar.

Re: GZIP all SAS datasets in a library.

Kurt_Bremser — Wed, 30 Apr 2014 06:40:19 GMT

If you have login access to the command line, there is no, _no_, NO!!!!! reason to prohibit xcmd. Stick that to your "administrators". x starts a shell that differs from a "normal" login shell only in its environment (working directory, some environment variables), but not in its privileges. Any havoc you can wreak on the system with the x command is even easier done from the CLI.

Instead of bringing forth strawman arguments, they should harden their system by proper granting of rights, having quotas, workload management and so on.

"My" users are reigned in by quota management in locations where they have write access (or I set up dedicated file systems that don't bother me if full), restricting the number of concurrent SAS sessions (so they don't cause excessive paging by running >30 SAS sessions at once, one of the idiots actually did that one time), and proper management of access rights to resources. I actually encourage them to use ssh and the x command to make their days easier, and I never had problems from that.

If your admins think that only some users should have xcmd (and CLI access) enabled, then they only need to define an additional workspace server that only the privileged users have access to.

Re: GZIP all SAS datasets in a library.

jakarman — Wed, 30 Apr 2014 08:50:09 GMT

Kumarz, Kurt is a little bit rude but he is right. Kurt and I could do an Alliance :smileydevil:. If I read his approach he is really engaged and helping his customers analists.

You have access to CLI (ssh) allowing all commands and not able to do the same with SAS. Silly/Crazy reasoning. Could by a Dilbert episode using humor as approach.

Why this blocking issue? There are a lot of policies at the OS level. I found an example at an professional organization. http://www.nsa.gov/ia/_files/os/redhat/rhel5-guide-i731.pdf Interesting is 2.3.1.4 as is telling users should not use the shell. That will work for routers or a DBMS with his own OS approach, but not when you want to use it for analytics in a way like Excel desktop approach. You need to convince and document analytics is different than a router approach. This should not be that difficult. The argument that this not common with SAS is a false one. Why SAS did this default setting? I want to keep polite, ....

It looks for me as SAS failed to understand the higher level requirements as set by reglautors (hipaa iso27k ITSM Cobit) and instead did some technical things as result of vague insinuations/questions.

What is to find about the xcmd? see: The case for XCMD privileges in SAS Enterprise Guide - The SAS Dummy Normal SAS usage in PC environments is quite common to use xcmds, their own desktop/data why should you bother them. At 9.1.3 Eguide was limited in the same way with xcmd and getting bad acceptance for that. That one is set open by default again at 9.3.

SAS is very open to integrate tools. With most of them you will find the XCMD access level is needed. As examples:

- using R with SAS/IML,

- Eminer user instructions,

- get through requirements of solutions you will also find many times the XCMD-requirement.

SAS-VA is marketed as web-based but the data-admins are needing xcmd (6.4).
For those third party components SAS is integrating they are promising to allow xcmd usage not needing that setting. That is bypassing their own check.

Why would you do all that trouble? Having a well designed OS layer security not allowing you to harm anybody else on the machine (sandboxed - non-shared parts - workload isolation cgroup) and when something is going wrong you can correct that yourself or somebody like Kurt will get to you.

This is asking a more professional approach at the OS level with this vision. Can be challenging.

Re: GZIP all SAS datasets in a library.

Kumarz — Sun, 11 May 2014 02:58:20 GMT

Kurt and Jaap,

I really appreciate both of yours' detailed response in this case. Kurt sounds little rude; but honestly he is 100% right. It didn't make sense to me as well when I asked the same question and they gave me all those security reasons.

As of now, I'm doing little more research before petitioning my administrator to grant as allow XCMD and remove other restrictions.

Since, I'm slowly moving towards Admin role; Can I ask you guys some specific questions via emails? I'm not sure if conversations like this (or future) should be part of this thread or not. Please let me know.

And like Jaap mentioned about creating an alliance with Kurt to help users like me; it would be great that we get direct guidance from industry experts like you...!!!.

Thank you very much.

Kumar.

Re: GZIP all SAS datasets in a library.

jakarman — Sun, 11 May 2014 06:16:15 GMT

Kumar,

I did recognize your issues as Kurt did, it is frustrating. Reading his answers/discussions you can recognize he understands SAS and the OS environment as a combination.

That is a needed combination that when lacking is giving that many problems. And make a guess how often this is wrong indeed.

For the arguments and going deeper into details about a real secured environment, I have given some links already.

"It looks for me as SAS failed to understand the higher level requirements as set by regulators (hipaa iso27k ITSM Cobit) and instead did some technical things as result of vague insinuations/questions."

We can do it openly. A lot of more people are having this kind of problems. It could be very interesting for SAS people also.

You are not the only one. I went to the OS details with SNLFAM1 in this one: Allowing Approved Permissions to Restricted Data

Building up the environment according the required security policies and that is not just following: a contradictionary way or humbug checklist statements as used by nitwits.

If it are details about persons or a company you are not wanting to have in a discussion thread you can mail me.

I should be easily to find ...(just google).

Re: GZIP all SAS datasets in a library.

Kurt_Bremser — Mon, 12 May 2014 07:27:10 GMT

Kumar,

if you are already moving towards an "admin" role:

- read, read, read. Since there are literally tons (in terms of paper, if printed) of SAS documentation out there, that is an absolute prerogative. I have found that posting the contents of an error message or a log in google with "SAS" prepended often returns a usable link to information immediately.

- get a solid grip on the operating system which is used for the SAS server. With CLI access, you may find you can solve lots of problems quickly using generic OS tools. Server OS's are VERY powerful, and the "read, read, read" is also valid there. Knowledge of the server OS puts you on an equal footing with those who run/administrate it. Large operations tend to produce people who avoid work by feeding bullshit to those that they consider to have no clue. Proving them wrong can be exhilarating and rewarding . I often find myself (in the admin role) close to falling into this trap, so I force myself to rethink WHY I have placed certain restrictions on users.

- real security for data has to be done at the OS level. Only there can you be sure that nothing untowards happens (think of your case: restricting access via metadata is useless if you can still use the CL to copy a file to your sasuser directory).

- if then you have problems that seem impossible to solve, or if you suspect that there may be more efficient solutions, don't hesitate to open threads here or in other places (stackexchange.com, stackoverflow.com etc). That way you tap into the cumulative wisdom of the whole _planet_.

Re: GZIP all SAS datasets in a library.

jakarman — Mon, 12 May 2014 07:44:56 GMT

This is one for everyone, excellent to be quoted:

"I often find myself (in the admin role) close to falling into this trap, so I force myself to rethink WHY I have placed certain restrictions on users."

Re: GZIP all SAS datasets in a library.

Kumarz — Wed, 14 May 2014 18:03:28 GMT

Yes.

I would agree to both of you on all the comments and suggestions. Especially Read, read, read and Mastering the OS part. I have been reading these (and other) blogs/threads and SAS documentations as much as I can and MOSTLY whenever I face a problem (like everyone else.......).

Of course its never ending process. But my confusion is how much is enough? Because in today's fast paced world nothing is enough and everything you learn/deliver is late when you are a new to an Admin role (or any other role).

So is there any suggestion (or any discussion on other forums/threads) that you guys can share with us? I mean, " How to become an efficient SAS Administrator : For a newbie or for a SAS programmer or for anyone?" If nothing is out there; then lets build/start something from this question.

It would be a great help for an aspiring SAS Administrator like me and others.

Any help is appreciated.

Kumar.