topic Matter with unzipping a big file in SAS Programming

Matter with unzipping a big file

LineMoon — Wed, 27 Apr 2016 20:27:30 GMT

Hello,

I have a big file .zip : Y:/M/N/BIG_File.zip, the size of big_file.zip is 6 GB.

So I want to unzip the file big_file.zip into a new folder : BIG_FILE

rep --> Y:/M/N/rep

I use this sas code

%let dir=Y:/M/N;

X cd &dir;

filename pipe "unzip BIG_FILE -d REP/BIG_FILE";

So I get this message :

unzip : cannot find BIG_FILE.zip, BIG_FILE.ZIP, BIG_FILE

So If i use the last code for a small file, the unzip works well.

So I need to unzip a big directory with more than 6 GB.

Thank you

Re: Matter with unzipping a big file

Astounding — Wed, 27 Apr 2016 20:52:24 GMT

Could it just be a spelling issue? Unix treats upper case as different than lower case.

Re: Matter with unzipping a big file

LineMoon — Wed, 27 Apr 2016 21:53:30 GMT

Thank you for your intereste.

No, it is not a matter of spelling,lowwer case, or upper case.

I have tested that.

the unzip works well for the small file, but not for a big file

Re: Matter with unzipping a big file

Reeza — Wed, 27 Apr 2016 22:16:40 GMT

Post your log a screenshot of your file in the path please.

Also, does it work if you do it via command line? If not, then it's not a SAS issue, it's an issue with the zip utility.

Re: Matter with unzipping a big file

Tom — Thu, 28 Apr 2016 02:50:38 GMT

What version of ZIP does your system have? You might not have support for large files. You need at least ZIP 3.0. Or you could try 7ZIP instead.

Re: Matter with unzipping a big file

Kurt_Bremser — Thu, 28 Apr 2016 04:40:38 GMT

Don't do a separate cd; instead use fully qualified filenames in the filename pipe. An external change directory will most probably have no effect on the current working directory of the SAS session.

Re: Matter with unzipping a big file

LineMoon — Thu, 28 Apr 2016 22:30:17 GMT

Thank you for your answer.

I will check that

Re: Matter with unzipping a big file

LineMoon — Thu, 28 Apr 2016 22:36:40 GMT

@Tom: Thank your message.

I think gzip. But as I hear from some people, sas used his zip.

So how can I identify the zip utilities on unix and version?

Re: Matter with unzipping a big file

Reeza — Thu, 28 Apr 2016 22:42:45 GMT

@LineMoon The method you're using to unzip is not a SAS method or procedure. It's using SAS to pass Unix Commands to your unix system. So whatever is on your box is what you have available.

It's another way of saying:

X "unzip blabhlbhafile";

If you want to know the utilities on unix ask your administrator.

Re: Matter with unzipping a big file

Kurt_Bremser — Fri, 29 Apr 2016 08:17:46 GMT

SAS has a built-in engine to unzip input files on the fly, which you can use in filename and infile statements. Things that you call externally, OTOH, depend on the commands supplied by the operating system.

By the way, you used

%let dir=Y:/M/N;

which looks like a Windows path, as UNIX does not use (and does not allow) drive letters.

Re: Matter with unzipping a big file

LineMoon — Sat, 30 Apr 2016 06:52:50 GMT

No , I want to say this

%let dir=Y/M/N;

No , it is not a windows path, it is a unix path.

in my code, I have used the correct path in unix %let dir= Y/M/N;

As I say before, the code does the job well for a small file, but it does not work for a big file ( more than 2 GB)

%let dir= Y/M/N;

X cd &dir;

filename pipe "unzip BIG_FILE -d REP/BIG_FILE";

Re: Matter with unzipping a big file

LineMoon — Sat, 30 Apr 2016 07:39:01 GMT

@KurtBremser

Thank you for your answer.

Please,

what is the impact of using "separate CD" in that ?

Can you explain " use fully qualified filenames in the filename pipe" ?

I observe that, If I do not use a cd, I get a strainge result for a small file, but it does not work for a big file

filename pipe "unzip V1/V2/V3/W1/W2 SMALL_FILE -d V1/V2/R/L/Small_FILE";

Is it possible to unzip a big file by sas without using Unix command.

Thank you again

Re: Matter with unzipping a big file

Reeza — Sat, 30 Apr 2016 10:57:43 GMT

Does the line work outside of SAS, can you unzip the file successfully at all?

Re: Matter with unzipping a big file

Kurt_Bremser — Sat, 30 Apr 2016 11:35:10 GMT

First of all, just read the documentation: http://support.sas.com/documentation/cdl/en/lestmtsref/68024/HTML/default/viewer.htm#n1dn0f61yfyzton1l2ngsa1clllr.htm

The ZIP Access Method only works for files compressed with WinZip.

What's so bad about using the right tool for unzipping files? UNIX features all tools necessary for uncompressing, so use them.

Regarding the "separate cd": UNIX keeps the current working directory in an environment variable (PWD) of the current process. When you use cd in a separate step, you might just change the PWD there, but (in a later step) end up in the initial working directory of the SAS process, as the environment of the subprocess where the cd was performed is lost. Therefore I recommend to use fully qualified path names for the files in the unzip command. A fully qualified path name is one that starts at the root directory - with a slash.

Never assume, take full control.

Re: Matter with unzipping a big file

Tom — Sat, 30 Apr 2016 13:14:05 GMT

Use the -v option to ask it what version of unzip you using. Contact your system administrator and ask them to give you access to a version that can support files larger than 2Gb.

data _null_;
   infile 'unzip -v' pipe;
   input;
   put _infile_;
run;

Re: Matter with unzipping a big file

LineMoon — Sat, 30 Apr 2016 15:49:35 GMT

Thank you for your answer.

I will check.

For information, the line command works for a small file in sas

Re: Matter with unzipping a big file

LineMoon — Sat, 30 Apr 2016 15:56:02 GMT

@Tom :Thank you for answer.

Re: Matter with unzipping a big file

LineMoon — Sat, 30 Apr 2016 16:29:12 GMT

@Kurt_Bremser

Thank you that's very kind from you

intersting, so Please :

- can we use the use command zip for all file with .zip extension : file.zip ?

If for some of files with extension .zip , we can not, so how can we identify, if the file was compressed by WinZip or not ?

- "A fully qualified path name is one that starts at the root directory" as you said

I have used that for a small file, it works, but I get a strainge folder in Small_FilE

If I use this code

filename pipe "unzip /V1/V2/V3/W1/W2 SMALL_FILE -d /V1/V2/R/L/Small_FILE";

I get this strainge directories :

Small_FilE//V1/V2/R/L/Small_FILE