topic Re: How To Correctly Use Pipe and WKUNZIP? in SAS Procedures

How To Correctly Use Pipe and WKUNZIP?

shellp55 — Thu, 31 Jan 2013 15:36:21 GMT

I am using SAS 9.3 (64 bit) on a Windows 7 computer.

I am trying to use pipe and wkunzip to read text files directly from the zip file. Based on the information that came with the wkunzip add on I used the syntax below but it isn't extracting from the text file portion. The log doesn't indicate any errors, just that "0 records were read from the infile testing". I am also curious if I need to identify the text file name because there could be many and/or could I use wildcards because they would all have the same naming convention i.e. 1234A, 1234B, 1234C etc.

Can someone please advise why this isn't working? Any and all assistance greatly appreciated.

filename testing pipe '"C:\Program Files\WinZip\wzunzip.exe" -vb F:\myfolder\Test.zip test.txt';

data work.import_test;
infile testing truncover LRECL = 5000 firstobs=10;

input @1 Prov   $1.
   @2 Inst   $4.
   @6 Fyear   $4.
   @10 Period   $2.
   @12 Batch   $2.
   @14 AbsNo      $3.
   @18 Coder   $2.
   @20 ChartNo   $10.
   @30 RegNo   $7.
   @37 SecChartNo $10.
   @47 MatNBChart $10.
   @57 HCN   $12.
   @69 Postal   $6.
   @75 Rescode   $7.
   @82 Sex   $1.
   @83 ProvHCN   $2.
   @85 RFP   $2.;
run;

Re: How To Correctly Use Pipe and WKUNZIP?

Stig_Eide — Thu, 31 Jan 2013 16:17:24 GMT

So, test.txt is inside Test.zip?

In that case, you could maybe do this:

filename testing pipe '"C:\Program Files\WinZip\wzunzip.exe" -vb F:\myfolder\Test.zip test.txt & type test.txt';

This would put the content of Test.txt to STDOUT and then the program would get the content of the file.

I guess, if you need to delete the file afterwards, you could add the delete command after the type command.

Hope this helps!

Stig

Re: How To Correctly Use Pipe and WKUNZIP?

HenrikDorf — Thu, 31 Jan 2013 16:30:59 GMT

The -vb option tells WZUNZIP to show the filenames in the zip archive and this is not what you want .

Change -vb to -c and the contents of the archived file will be output , unfortunately sorrounded by extranous information from the wzunzip utility:

I think that WZUNZIP does not support piping.

197 filename nziip pipe 'wzunzip -c C:\TEMP\TESTZIP.zip test.txt' ;

198

199 data _null_ ;

200 infile nziip ;

201 input ;

202 put _infile_ ;

203 run;

NOTE: The infile NZIIP is:

Unnamed Pipe Access Device,

PROCESS=wzunzip -c C:\TEMP\TESTZIP.zip test.txt,

RECFM=V,LRECL=256

WinZip(R) Command Line Support Add-On Version 3.1 (Build 8519)

Zip file: C:\TEMP\TESTZIP.zip

============================

Unzipping test.txt to con

-------------------------

1:,Count,Date,Time,,c:\pagefile.sys,C:\temp2,C:\temp

2:,0,21042012,202204,0,12667846656,0,273826763

2:,1,21042012,202205,0,12667846656,0,273826811

2:,2,21042012,202206,0,12667846656,0,273826859

2:,3,21042012,202207,0,12667846656,0,273826907

2:,4,21042012,202208,0,12667846656,0,273826955

Stderr output:

Searching...

NOTE: 14 records were read from the infile NZIIP.

The minimum record length was 0.

The maximum record length was 70.

NOTE: DATA statement used (Total process time):

real time 0.13 seconds

cpu time 0.00 seconds

If you use another unzip utility that supports PIPING , the result will be as you want:

205 filename zipfile pipe 'unzip -p C:\TEMP\TESTZIP.zip test.txt' ;

206

207 data _null_ ;

208 infile zipfile ;

209 input ;

210 put _infile_ ;

211 run;

NOTE: The infile ZIPFILE is:

Unnamed Pipe Access Device,

PROCESS=unzip -p C:\TEMP\TESTZIP.zip test.txt,

RECFM=V,LRECL=256

1:,Count,Date,Time,,c:\pagefile.sys,C:\temp2,C:\temp

2:,0,21042012,202204,0,12667846656,0,273826763

2:,1,21042012,202205,0,12667846656,0,273826811

2:,2,21042012,202206,0,12667846656,0,273826859

2:,3,21042012,202207,0,12667846656,0,273826907

2:,4,21042012,202208,0,12667846656,0,273826955

NOTE: 6 records were read from the infile ZIPFILE.

The minimum record length was 46.

The maximum record length was 52.

NOTE: DATA statement used (Total process time):

real time 0.10 seconds

cpu time 0.00 seconds

UNZIP.exe can be found at various open source places on the internet - just make sure that PIPING (-p) is supported.

Cheers

Henrik Dorf

Denmark

Re: How To Correctly Use Pipe and WKUNZIP?

shellp55 — Thu, 31 Jan 2013 20:40:20 GMT

Thanks for the replies but still not working in that I get the message 0 files read but no error messages. Henrik according to the SAS knowledgebase, pipe and wkunzip are supposed to work together even though one of the messages showing up is "unnamed pipe access device".

Re: How To Correctly Use Pipe and WKUNZIP?

HenrikDorf — Thu, 31 Jan 2013 21:35:54 GMT

When i say that WKUNZIP dos not support pipe - i mean that it is not able to exclude messages and trademarks in the output.

I can see that the infile option "firstobs=10" will take care of the first 8 lines of information from WKUNZIP and the first dataline which presumably contains variable names.

Personally i prefer the clean input from UNZIP with no messages from the unzip utility.

Unnamed pipe access device is a SAS concept since you can define named and unnamed pipes in filename statements.

And finally for solving the problem:

The LRECL=5000 seems to have impact in my test - so I would suggest that you remove this parameter to see if it helps you too.

Henrik

Re: How To Correctly Use Pipe and WKUNZIP?

HenrikDorf — Thu, 31 Jan 2013 21:54:12 GMT

I just realized that truncover will handle the problems that could occur if LRECL does not match the true recordlength.

Sorry ...this will not solve anythng

It i strange that you have the message " 0 files read"

To see the messages from WKUNZIP you could write the program like this:

data work.import_test;
infile testing truncover LRECL = 5000 ;

IF _N_<11 then do;

input ;

putlog _infile_;

return;

end;

input @1 Prov   $1.
   @2 Inst   $4.
   @6 Fyear   $4........

Henrik

Re: How To Correctly Use Pipe and WKUNZIP?

DBailey — Thu, 31 Jan 2013 22:27:14 GMT

If your system administrators allow...you might could perform the extract via the X command and then process the file:

X '"C:\Program Files\WinZip\wzunzip.exe" -vb F:\myfolder\Test.zip test.txt';

filename testing "F:\myfolder\test.txt" LRECL = 5000;

Re: How To Correctly Use Pipe and WKUNZIP?

shellp55 — Thu, 31 Jan 2013 22:51:30 GMT

Thanks for the responses.

Henrik: there was a message of "NOTE: DATA STEP stopped due to looping" so can't it just be an if statement?

DBailey: I assume that this is to unzip the file to the folder and from there I can input it? If so, that didn't work because I get the error message "ERROR: Physical file does not exist, F:\myfolder\test.txt."

I got the following to work in that it shows in the log what the contents are but I want to read and extract/manipulate the data while keeping it within the zip file. Is that not possible? Even if I extract it, manipulate it and then delete the extraction (leaving the zip file as is), is that possible? Also, I haven't even gotten to password protection yet which I was hoping I could include or there is no point in this exercise.

I got this to work to show the output in the log:

filename testing pipe '"C:\Program Files\WinZip\wzunzip.exe" -c F:\myfolder\Test.zip test.txt';

data _null_ ;
infile testing;
input ;
put _infile_ ;
run;

Thanks for any and all assistance!

Re: How To Correctly Use Pipe and WKUNZIP?

Stig_Eide — Thu, 31 Jan 2013 23:32:12 GMT

There is a SUGI paper that discusses different strategies to read files inside compressed archives:

http://www2.sas.com/proceedings/sugi31/155-31.pdf

maybe that can help?

Stig

Re: How To Correctly Use Pipe and WKUNZIP?

DBailey — Thu, 31 Jan 2013 23:43:36 GMT

So that suggests either you can't use the X commands (unlikely if you didn't get an error) or the unzip syntax is incorrect. If you just run the x command and then go look in the F folder, does the file exist?

Re: How To Correctly Use Pipe and WKUNZIP?

shellp55 — Fri, 01 Feb 2013 00:34:19 GMT

Thanks for sticking with me!

Stig: thanks, yes I've looked at this and can't get that to work either. I read in another post where someone contacted the author of the paper and found he didn't test all of his code.

DBailey: I am the only person on my computer but I have an admin and user profile. I ran the program from my admin profile just in case there were issues with my not being able to use the x command and there was the error message about no physical file existing. So, using either profile, no file shows up in the F folder suggesting the syntax for unzip is incorrect.

Any other thoughts?

Re: How To Correctly Use Pipe and WKUNZIP?

shellp55 — Fri, 01 Feb 2013 01:16:12 GMT

Okay, I don't know why it's working this time but I revisited all the previous information I had including the blurb from the SAS knowledgebase site. I used what they had like I did before but this time it worked.

I also found the syntax to use the password function (use -s and your password right after - in the example below my password is 1234). So the following works and I thank all who tried to help me!

filename testing pipe '"C:\Program Files\WinZip\wzunzip.exe" -s1234 -o -c F:\myfolder\test.zip' ;

data test_import ;
infile testing firstobs=10 truncover LRECL=5000;

input @1 Prov   $1.
   @2 Inst   $4.
   @6 Fyear   $4.
   @10 Period   $2.
   @12 Batch   $2.
   @14 AbsNo      $3.
   @18 Coder   $2.
   @20 ChartNo   $10.;

run;

Note that as was alluded to by DBailey (and SAS knowledgebase), you have to select firstobs=10 to bypass the code from WinZip. I think another issue for me was that I wasn't including the firstobs statement first. Note that I don't have to cite the files within the zip which is great because I have lots. If I wanted to exclude some I could use the -e syntax and I would imagine I can limit by file extension versus name if I wanted to.

I am so very pleased to get this darn thing working! Thanks again to anyone who responded!

Re: How To Correctly Use Pipe and WKUNZIP?

DBailey — Fri, 01 Feb 2013 13:25:26 GMT

what happens if you open a command window and run the command outside of sas?

"C:\Program Files\WinZip\wzunzip.exe" -vb F:\myfolder\Test.zip test.txt