Filename Pipe X Command -- Limitations?

[jimbarbour]

I'm running SAS 9.4 M6 on a 64 bit Windows Server 2016 machine.  I'm using WnZip.exe to zip files as part of a FILENAME PIPE.  

For example:

FILENAME	WinOpSys	PIPE	"&ZipPgmPath. -a ""&Zip_File."" ""&Full_File."""	LRECL=&LRECL;

So long as the files are below about 8 Gb in (unzipped) size, all is well and good.  Log excerpt:

       +-----------------------------------------------------------------------------------------------------------------+
NOTE:  | Start Zip at 2020-03-02 12:33:42.20.                                                                            |
       | WinZip(R) Command Line Support Add-On Version 4.0 64-bit (Build 10562)                                          |
       | Copyright (c) 1991-2013 WinZip International LLC - All Rights Reserved                                          |
       |                                                                                                                 |
       | Updating plans.sas7bdat                                                                                         |
       | Total bytes=4653056, Compressed=381416 -> 92 percent savings.                                                   |
       |                                                                                                                 |
       | Replacing old Zip file "I:\Commercial\monthly_data\Cornerstone\archive\Stg\Plans_Archive_2020-03-02_1203.zip".  |
       |                                                                                                                 |
       | End Zip   at 2020-03-02 12:33:42.93.                                                                            |
       | Return code from zipping Plans.sas7bdat = 0.                                                                    |
       +-----------------------------------------------------------------------------------------------------------------+

But if the files are somewhere over 8 to 10 Gb, the Zipping just hangs.  It neither completes nor fails.  There's no message, there's no error code or SYSRC posted, and the job just runs and runs.  

The same file when submitted manually from the command line zips just fine.

Likewise, if the file is submitted as an "X" command but not as part of a FILENAME PIPE, the file zips just fine.  The only difference in the runs that work vs. those that do not is the FILENAME PIPE.

What are the limits on FILENAME PIPE?  And are there workarounds available? I've tried increasing memsize, but that doesn't seem to help.  I can see on repeated attempts to zip the same file, that the intermediate zip work file freezes at the same size repeatedly.

Anyone have any light to shed on this or suggestions?

Thank you,

Jim

P.S.  And, yes, I can just use the "X" command exclusive of a FILENAME PIPE, but I like having access to STDOUT on a line by line basis as WinZip processes, and I like having STDERR go to the SAS log.  Therefore, FILENAME PIPE is hardly a "do or die" necessity, but I would strongly prefer FILENAME PIPE if I can get it to work and at a minimum would like to know what's going on and why it's different than just an "X" command alone.

[Patrick]

It feels like a timeout issue. What SAS version and OS are you on. And how do you run your code?

If you're on a rather recent version then have also a look into the filename ZIP engine.

[jimbarbour]

Patrick,

A time out issue?  Possibly although I would expect the job to end if it somehow timed out.  The job neither ends nor errors.  Perhaps that's a time out, but if so I'm disappointed that SAS isn't ending and returning messages and setting SYSRC.

I'm running SAS 9.4 M6 on a 64 bit Windows Server 2016 machine.  I'm using WnZip.exe to zip files as part of a FILENAME PIPE.   I have a code excerpt in my original post.  We have WinZip(R) Command Line Support Add-On Version 4.0 64-bit (Build 10562) Copyright (c) 1991-2013 WinZip International LLC. The executable is WinZp.exe.

As for WinZp.exe, it runs fine with the exact same files -- so long as I run not as part of a FILENAME PIPE.  

There's something about running as part of a FILENAME PIPE that is causing things to hang.

Jim

[Patrick]

Timeout is just a guess. Googling I haven't really found much in this direction except for SAS9.1.3

But.... If using a recent SAS version I'd suggest to not use a pipe at all but the filename ZIP engine.

@ChrisHemedinger wrote here quite a few blogs about it.

[jimbarbour]

@Patrick,

Hmmm.  That's not a bad idea, using SAS itself to process the zip files.

In this case, I'm just zipping some files, and I don't need to actually look at the contents, so I'm thinking that SAS may be a bit less efficient than just using wzzip.exe directly.

What I'm doing is backing up a directory a few days before a monthly production job will update all the files.

I have a macro, %archive_dir, that gets all the names of sas7bdat or sas7bndx files and passes them to another macro, %archive_data, that calls wzzip.exe via an "X" command to zip the files into a Zip Archive that is saved into a sub directory.  

I think wzzip.exe is going to be a lot more efficient for this purpose than reading each file line by line with a SAS program and then writing that file out with the ZIP parameter on a Libname.

The suggestion however is a good one, and I have certainly used the ZIP parameter on a libname in cases where I need to actually work with the data at the record level and then zip it.  And of course, generally any reference to @ChrisHemedinger 's blog is well taken.  🙂

Jim

[Tom]

Make sure that WINZIP is not trying to ask you a question.

Also where is WINZIP writing the ZIP file?  There a probably options you can set to tell it where to write the temporary file(s) it needs to complete the action.  Most likely the default location is different when being called by your SAS process.

[jimbarbour]

@Tom,

That's a good point to double check that wzzip.exe isn't prompting me for something.  However, when I execute the exact same commands via a stand alone X command or manually from a DOS prompt, there is no dialog, so I'm not thinking at this juncture that that is the issue.  In those cases where I use the X command not stand alone but embedded in a FILENAME PIPE, I do capture both STDOUT and STDERR, so I should be able to see any dialog -- but I am not.

Wzzip.exe is writing to a sub-directory named Archive.  In my macro, I call another macro, cd.  "cd" stand for Change Directory, and it changes the SAS working directory via the DLGCDIR function.  I believe the macro is working correctly because 1) I can see a note in the SAS log (see below) and 2) because I can see the intermediate work files as Wzzip.exe processes the sas7bdat/sas7bndx files.

NOTE: The current working directory is now "I:\Commercial\monthly_data\Cornerstone\archive\Stg".

If it were helpful, I could post a screen shot of the "hung" intermediate files.  I can watch them incrementally grow until at some point they freeze and the SAS job freezes as well.

To your point, it could be that Wzzip.exe is expecting some kind of response, but given that when the same commands are executed manually via a DOS prompt or via a stand alone X command there is no dialog, I'm not thinking that's the issue.  I may however try doing a redirect of STDERR to the SAS log per @ChrisNZ 's suggestion, just to double check.

Jim

[Tom]

You should definitely figure out how to tell Winzip where to write its temporary files.

I sounds like you are zipping multiple files into one ZIP file.  So what it most likely is doing is reading the files one by one, compressing them and then adding them to the "archive", the ZIP file.  So that is probably why you see it working and making progress, then it hits some large file and it does not have enough room to create the compressed version of that file and is getting hung.

[Reeza]

Just some quick confirmations:

1. Does the command run in the command line, not via pipe?

2. Does the command run, via WinZip manually - You've confirmed this works fine. 

3. Is there enough space for the file to unzip or is there any possibility you're running out of space somehow?

4. How are you capturing results to ensure there's no error being passed back, you didn't include that portion of your code. 

Is it different than what you get from #1?

[jimbarbour]

@Reeza,

I've run the WnZip.exe in question three ways:

1.  As an X command alone.

2.  As an X command embedded in a FILENAME PIPE

3.  Manually from a DOS prompt (command line).

All three work equally well unless the file is larger than about 8Gb in which case number two, above, fails to complete; my job just gets stuck and goes nowhere.

I have 3.8 Tb free on my drive.  I do not suspect a disk space shortage, but it is good that you ask since that must be considered.  In this case, zipping a 8 Gb file should not result in a space issue on a drive with multiple terabytes free.

When I run the WnZip.exe via an X command that is part of a FILENAME PIPE, error output (STDERR) is written automatically to the SAS log, but in this case the job neither completes nor fails, and there is no error message.  The job only "hangs" and goes nowhere.

When I run WinZp.exe as an X command that is not part of a FILENAME PIPE, I have had no failures, so there is no error message. 

So also is there no error message when running manually from a DOS prompt.  Everything works fine.

Jim

[Tom]

Do you have an admin for the WIndows Server machine that can check what is happening?
Can you switch to 7z instead?  I haven't used Winzip in 10+ years.

[Tom]

Your terminology is a little confusing. SAS has an X command for running operating system commands.  It also has the SYSTEM command.  You can also use the PIPE filename engine to run an operating system command.  You use the PIPE engine either in the a FILENAME statement that defines an fileref or directly in your FILE statement.

When you use the FILENAME statement it just defines the fileref, but it does not actually try to execute the command until you try to use the file.  In your case when you try to READ from the file.  Normally that would be in a data step, but there are many other commands/functions in SAS that can read from a file.

You did not show in the code snippet you posted how you are trying to read the file.  Are you sure that step is not getting into an infinite loop because of the different messages that WINZIP is sending for the large file?

Please clarify that you know where WINZIP is writing its TEMPORARY files for both your SAS driven calls and your interactive calls.  I seem to remember that used some Windows environment variable like, %TEMP%, by default.  I definitely in the past had to override that because our default setup on Windows was pointing to a disk that did not have enough space for WINZIP to make the temporary files it needed.

[jimbarbour]

@Tom  and @ChrisNZ ,

Let me post some additional code.

Here is the essential code, in my macro.  I can post the entire macro if that's helpful, but a lot of the code is housekeeping.

First, there is a FILENAME PIPE with an X command embedded in it.  The X command executes wzzip.exe.  

FILENAME	WinOpSys	PIPE		"&ZipPgmPath. -a ""&Zip_File."" ""&Full_File.""";

&ZipPgmPath is the fully qualified zip program, wzzip.exe.

%LET	ZipPgmPath						=	E:\%STR("")Program Files%STR("")\WinZip\wzzip.exe;

&Zip_File is the fully qualified Zip Archive, i.e. this is the path and file name of the .zip file I am creating.

&Full_File is the fully qualified sas data that I want to zip into the Zip Archive.

Pretty straightforward, and when executed alone (not part of a FILENAME PIPE) or manually from a DOS prompt, works fine.  It also works very well with file sizes less than 8 Gb.

Then I read the STDOUT coming back from wzzip.exe using a DATA step:

DATA	_NULL_;
	LENGTH	_Curr_DateTime	$23;
	LENGTH	_Line			$255;
	LENGTH	_RC				4.;
	LENGTH	_First_Time		3.;
	LENGTH	_Total_Bytes	3.;

	RETAIN	_First_Time		1;
	RETAIN	_Total_Bytes	0;

	IF	_First_Time										THEN
		DO;
			_First_Time					=	0;
			PUTLOG	"&Nte2  %Format_Dashes(&Width)";
*						_Curr_DateTime				=	TRANWRD(PUT(DATETIME(), E8601DT23.2), 'T', ' ');
*						_Line						=	RESOLVE(CAT('&Nte1  %Format_Line(Start Zip at ', STRIP(_Curr_DateTime), '., &Width)'));
*						PUTLOG	_Line;
		END;

	INFILE	WinOpSys	TRUNCOVER	
						END				=	_Zip_Process_Complete
						LENGTH			=	_SAS_Length
						;
	INPUT	_WinZip_Msg	$VARYING32766.		_SAS_Length;

	_Line								=	RESOLVE(CAT('&Nte2  %Format_Line(%STR(', STRIP(_WinZip_Msg), '), &Width)'));
	PUTLOG	_Line;

	IF	INDEX(_WinZip_Msg, 'Total bytes=')				THEN
		DO;
			&Cmnt	PUTLOG	"&Nte2  | 'Total bytes=' has been found";
			_Total_Bytes				=	1;
		END;

	IF	_Zip_Process_Complete							THEN
		DO;
			IF	NOT	_Total_Bytes						THEN
				DO;
					PUTLOG	"&Warn2  %Format_Dashes(&Width)";
					_Line				=	RESOLVE(CAT('&Warn1  %Format_Line(&sQuote.Total bytes=&sQuote. not returned by WinZip, &Width)'));
					PUTLOG	_Line;
					_Line				=	RESOLVE(CAT('&Warn2  %Format_Line(Results are suspect.  Check zip archive., &Width)'));
					PUTLOG	_Line;
					PUTLOG	"&Warn2  %Format_Dashes(&Width)";
					_RC					=	INPUT(SYMGET('SysRC'), 4.);
					IF	_RC				<	4			THEN
						DO;
							&Cmnt	PUTLOG	"&Nte2  Setting SYSCC and SYSRC to 4";
							CALL	SYMPUTX('SYSCC', '4', 'G');
							CALL	SYMPUTX('SYSRC', '4', 'G');
						END;
				END;
			_Curr_DateTime				=	TRANWRD(PUT(DATETIME(), E8601DT23.2), 'T', ' ');
			_Line						=	RESOLVE(CAT('&Nte2  %Format_Line(End Zip at ', STRIP(_Curr_DateTime), '., &Width)'));
			PUTLOG	_Line;

			_RC							=	INPUT(SYMGET('SysRC'), 4.);

			IF	_RC						=	0			THEN
				DO;
					PUTLOG	"&Nte2  %Format_Line(Return code from zipping &File_Name = 0., &Width)";
					PUTLOG	"&Nte2  %Format_Dashes(&Width)";
					PUTLOG	"&Nte2  ";
				END;
			ELSE
				DO;
					PUTLOG	"&Err2  %Format_Dashes(&Width)";
					_Line				=	RESOLVE(CAT('&Err1  %Format_Line(Return code from zipping &File_Name = ', STRIP(PUT(_RC, 4.)), '., &Width)'));
					PUTLOG	_Line;
					PUTLOG	"&Err2  %Format_Dashes(&Width)";
					PUTLOG	"&Err2  ";
				END;
		END;
RUN;

The use of %Format_Dashes and %Format_Line just takes whatever is coming back from wzzip.exe and formats into a "box" in the SAS log:

2 The SAS System                                                                                 11:14 Tuesday, March 3, 2020
       +-----------------------------------------------------------------------------------------------------------------+
       | Start Zip at 2020-03-03 11:01:40.25.                                                                            |
       | WinZip(R) Command Line Support Add-On Version 4.0 64-bit (Build 10562)                                          |
       | Copyright (c) 1991-2013 WinZip International LLC - All Rights Reserved                                          |
       |                                                                                                                 |
       | Adding plans.sas7bdat                                                                                           |
       | Total bytes=4653056, Compressed=381416 -> 92 percent savings.                                                   |
       |                                                                                                                 |
       | Copying Zip file "I:\Commercial\monthly_data\Cornerstone\archive\Stg\Plans_Archive_2020-03-03_1113.zip".        |
       |                                                                                                                 |
       | End Zip at 2020-03-03 11:14:20.15.                                                                              |
       | Return code from zipping Plans.sas7bdat = 0.                                                                    |
       +-----------------------------------------------------------------------------------------------------------------+

I can run the X command and create an actual physical text file with the output from wzzip.exe and then use the same DATA step to read in that physical text file, but then my "Start Zip" message is wildly inaccurate, and I suspect my check of SYSRC may not work since the data step starts after wzzip.exe is completely finished.  Having the "Start Zip" message isn't essential, but I'd like to be able to catch any issues that SYSRC might reveal.  We're assuming we have a reliable backup and are going to start updating files after this, so we really want to know if something goes wrong.  I may be being a little paranoid here.  In the data step I am checking for the text "Total bytes=" which appears to be a pretty reliable indicator that the zip has worked correctly.

Jim

[ChrisNZ]

Just an observation (probably unrelated):

I would put the quotes around the whole path. I haven't seen the syntax you use before.

Also because there are two quotes, you don't need %str.

%let ZipPgmPath	= "E:\Program Files\WinZip\wzzip.exe";

Also, what happens if you use the data step example I provided?

Just trying to isolate the root cause here. Is it the INFILE's inner workings or is it your data step's logic, even though it simple enough.

[Tom]

I would eliminate all of that complexity. It will let you test more easily.  It will also make your program easier to use.

Don't put the quotes into your macro variables, add them when you need them.  

Instead of using CD just specify the fully qualified name of the ZIP file.

Instead of trying to re-direct the messages to a text file just let them come to your SAS program.

Try setting the TEMP environment variable to point to a place where you know there is enough space.  Perhaps the same place as you are writing the files.  I could not find real documentation on-line but this screen shot does show both TEMP and TMP environment variables.  https://support.winzip.com/hc/en-us/articles/115011955108-The-PATH-environment-variable You might try setting both.

%let zippgmpath = e:\program files\winzip\wzzip.exe;
%let free_space = d:\my_big_disk\temp ;
%let zip_path = c:\my_big_disk\archives;
%let zipfile = archive_2020_03_03.zip;
%let full_file = "c:\my_other_file\*.sas7bdat" ;

data _null_;
  infile %sysfunc(quote(
set TEMP=&free_space
; "&zippgmpath" -a "&zip_path\&zipfile" &full_file 
)) pipe ;
  input;
  put _infile_;
run;