SAS and winzip

Reply
Super Contributor
Posts: 371

SAS and winzip

[ Edited ]

Hello experts ,

I want to use sas with winzip  9 commands under windows on parallel.

I want to zip a many files ( more than 60) in parallel, I can zip them, but in long time.

Please, which commandes, I can use to accelerate the zipping.

Thank you

Super User
Posts: 19,822

Re: SAS and winzip

You'll only be using SAS to pass the system commands. Since this is no longer really a SAS question you're better off posting in a different forum, most likely StackOverflow.

 

Winzip has a command line guide and a first suggestion is you can change the compression rate.

 

Make sure the system commands work via command line before using SAS to execute them. 

 

http://stackoverflow.com/questions/tagged/winzip

 

 

Super Contributor
Posts: 371

Re: SAS and winzip

@Reeza:

Thank you for your message.

Please, why "change the compression rate." and how ?

 

Super User
Posts: 19,822

Re: SAS and winzip

Compression size is how small it makes the file. If you allow it to be a bit larger it won't take as much time. 

 

Read the winzip docs for the command or post on a winzip related forum. 

Super User
Posts: 11,343

Re: SAS and winzip

You might show us what you are doing that takes a long time.

 

Are you creating 60 output zip files or attempting to put 60 files into one zip?

It has been a long time since I used commandline Winzip but if you are putting multiple files into a single Zip file then I think you can use wild cards to indicate the source files.

 

Also how big are these files? Size does matter when it comes to just about any file manipulation.

And if the source, destination or both locations are on network drives you may have some other issues with IO through put in general that changing how the zip commands are executed may not effect.

Super Contributor
Posts: 371

Re: SAS and winzip

[ Edited ]

Thank you for your message.

 "Are you creating 60 output zip files or attempting to put 60 files into one zip?"

I am creating more than 60 output  folder  zip, but many files in every folder ( R1/t1.txt,    tn.txt), will be R1.zip

If i go from 30 to 60 in parallel, I multiply the zipping time  by 2. It become very long.

 

Also how big are these files?

It is about 1.6 GB before the zip.

Thank you

 

Super User
Posts: 7,809

Re: SAS and winzip

Parallelization only makes sense if you have the resources. Once you run out of hardware to run all the threads, context switching will cause more overhead than you get in performance from running parallel. Once you know your sweet spot, run according blocks of files consecutively.

Depending on your storage (spinning rust vs. SSD), parallel compression may cause disk contention and mandate consecutive processing.

Also, what is "long" for you? A single 1.6 GB file should take ~20 seconds with gzip.

---------------------------------------------------------------------------------------------
Maxims of Maximally Efficient SAS Programmers
Super Contributor
Posts: 371

Re: SAS and winzip

Posted in reply to KurtBremser

@KurtBremser : Thank you for your answer.

20 minutes is realy good...when I use the vbs program  to zip, I get 4  to 7 minutes.

So, I must have to get high compression ratio and very past compression.

Super User
Posts: 19,822

Re: SAS and winzip

So why not call the VBS program from SAS?

 

If it absolutely has to be a SAS solution, use SAS to create the VBS script and then execute it.

Super Contributor
Posts: 371

Re: SAS and winzip

So why not call the VBS program from SAS?

In some case, I can do that, if a sas server connected to my folders to zip  in server b.

In other case, I can not do that, if a  sas server has not acces to the server c. That's why, I try to use the solution in the right place.

Thank you

Super User
Posts: 7,809

Re: SAS and winzip


LineMoon wrote:

@KurtBremser : Thank you for your answer.

20 minutes is realy good...when I use the vbs program  to zip, I get 4  to 7 minutes.

So, I must have to get high compression ratio and very past compression.


Do you work on networked drives?
4 minutes to compress < 2 GB is ridiculous.

My performance of 20 seconds was achieved with gzip on a 10 year old p520

---------------------------------------------------------------------------------------------
Maxims of Maximally Efficient SAS Programmers
Super Contributor
Posts: 371

Re: SAS and winzip

Posted in reply to KurtBremser
Hi KurtBremser, Yes, I work the network drive, why ? I agree 4 minutes is not a good solution for 2 GB. I do not understand well , what you mean by "My performance of 20 seconds was achieved with gzip on a 10 year old p520" Did you publish a document about that ? Thank you again
Super User
Posts: 7,809

Re: SAS and winzip

Compressing files on a networked drive is, to be polite, a not very intelligent thing to do. You force concurrent read and write operations over the network.

Best solution is to do the compression on the fileserver itself. Second best: compress to a local file and copy that back to the server later.

 

My measurements come from experience with my SAS server, which is a IBM pSeries p520. But I use proper SAN storage with a minimum throughput of 100 MB/sec.

---------------------------------------------------------------------------------------------
Maxims of Maximally Efficient SAS Programmers
Super User
Posts: 19,822

Re: SAS and winzip

Are you creating the files from SAS? If so can you use ODS Package ZIP?

This is available as of SAS 9.2

 

http://support.sas.com/resources/papers/proceedings09/018-2009.pdf

Super User
Posts: 5,431

Re: SAS and winzip

Again, this has little to do with SAS and best asked elsewhere.
Data never sleeps
Ask a Question
Discussion stats
  • 14 replies
  • 546 views
  • 3 likes
  • 5 in conversation