Most efficient way to read Zip Txt files

Reply
Super Contributor
Posts: 418

Most efficient way to read Zip Txt files

Hello all. I have many (tens of thousands) of files in Zip format that I have to read into a database. ON this note, does anyone know the most efficient way to read in Zip files into sas?

I have a macro that loops through and runs WinZip, and un-zips the files, however that alone will take several days of processing time. Is there a methodology to read in zip files without compressing them that is faster than simply un-zipping the files first and then reading them in?

Computer Information

Have Winzip

Windows Xp computer.

Let me know if you need any further information on the system!


Brandon

Super User
Super User
Posts: 7,074

Re: Most efficient way to read Zip Txt files

Posted in reply to Anotherdream

Get the command line utility for WinZip off their web site.

Then you can read the files as they are expanded using a pipe.

Probably will not save much time, other than by eliminating the need for you to respond to prompts, but it should eliminate the need to have space for all of he expanded text files.

I forget the name of the Windows command line program for Winzip so I will use the unix command (unzip) in the following example.

data dataset1;

   infile "unzip -p MyFile.zip MyFile.txt" pipe ;

   input name $ age sex;

run;

Super Contributor
Posts: 418

Re: Most efficient way to read Zip Txt files

Thank you Tom, I will attempt this.

Does the command line utility basically work the same way that Winzip does (with  the exception of not needing to save the file as you mentioned!), in that no matter what you have to un-zip a file in order for SAS to read it in?


So basically there is no cool trick that allows you to read in a zipped file without the work of un-zipping it in some manner.

Thanks again, you're always very helpful.

Brandon

Trusted Advisor
Posts: 2,116

Re: Most efficient way to read Zip Txt files

Posted in reply to Anotherdream

pipe on Windows is less smart that in *nix.  It actually creates an unzipped temporary separate copy and then reads that.  It mostly saves the manual cleanup.

BTW, for text files, the Windows disk compression is about as efficient as WinZip.

Community Manager
Posts: 2,955

Re: Most efficient way to read Zip Txt files

Posted in reply to Anotherdream

In SAS 9.4, you can use FILENAME ZIP.  Here's an example with the details:

Reading and updating ZIP files with FILENAME ZIP - The SAS Dummy

Chris

Ask a Question
Discussion stats
  • 4 replies
  • 1463 views
  • 0 likes
  • 4 in conversation