topic Re: Filename Pipe X Command -- Limitations? in SAS Programming

Filename Pipe X Command -- Limitations?

jimbarbour — Tue, 03 Mar 2020 00:12:54 GMT

I'm running SAS 9.4 M6 on a 64 bit Windows Server 2016 machine. I'm using WnZip.exe to zip files as part of a FILENAME PIPE.

For example:

FILENAME	WinOpSys	PIPE	"&ZipPgmPath. -a ""&Zip_File."" ""&Full_File."""	LRECL=&LRECL;

So long as the files are below about 8 Gb in (unzipped) size, all is well and good. Log excerpt:

       +-----------------------------------------------------------------------------------------------------------------+
NOTE:  | Start Zip at 2020-03-02 12:33:42.20.                                                                            |
       | WinZip(R) Command Line Support Add-On Version 4.0 64-bit (Build 10562)                                          |
       | Copyright (c) 1991-2013 WinZip International LLC - All Rights Reserved                                          |
       |                                                                                                                 |
       | Updating plans.sas7bdat                                                                                         |
       | Total bytes=4653056, Compressed=381416 -> 92 percent savings.                                                   |
       |                                                                                                                 |
       | Replacing old Zip file "I:\Commercial\monthly_data\Cornerstone\archive\Stg\Plans_Archive_2020-03-02_1203.zip".  |
       |                                                                                                                 |
       | End Zip   at 2020-03-02 12:33:42.93.                                                                            |
       | Return code from zipping Plans.sas7bdat = 0.                                                                    |
       +-----------------------------------------------------------------------------------------------------------------+

But if the files are somewhere over 8 to 10 Gb, the Zipping just hangs. It neither completes nor fails. There's no message, there's no error code or SYSRC posted, and the job just runs and runs.

The same file when submitted manually from the command line zips just fine.

Likewise, if the file is submitted as an "X" command but not as part of a FILENAME PIPE, the file zips just fine. The only difference in the runs that work vs. those that do not is the FILENAME PIPE.

What are the limits on FILENAME PIPE? And are there workarounds available? I've tried increasing memsize, but that doesn't seem to help. I can see on repeated attempts to zip the same file, that the intermediate zip work file freezes at the same size repeatedly.

Anyone have any light to shed on this or suggestions?

Thank you,

Jim

P.S. And, yes, I can just use the "X" command exclusive of a FILENAME PIPE, but I like having access to STDOUT on a line by line basis as WinZip processes, and I like having STDERR go to the SAS log. Therefore, FILENAME PIPE is hardly a "do or die" necessity, but I would strongly prefer FILENAME PIPE if I can get it to work and at a minimum would like to know what's going on and why it's different than just an "X" command alone.

Re: Filename Pipe X Command -- Limitations?

Patrick — Wed, 04 Mar 2020 00:39:55 GMT

It feels like a timeout issue. What SAS version and OS are you on. And how do you run your code?

If you're on a rather recent version then have also a look into the filename ZIP engine.

Re: Filename Pipe X Command -- Limitations?

jimbarbour — Tue, 03 Mar 2020 00:59:34 GMT

Patrick,

A time out issue? Possibly although I would expect the job to end if it somehow timed out. The job neither ends nor errors. Perhaps that's a time out, but if so I'm disappointed that SAS isn't ending and returning messages and setting SYSRC.

I'm running SAS 9.4 M6 on a 64 bit Windows Server 2016 machine. I'm using WnZip.exe to zip files as part of a FILENAME PIPE. I have a code excerpt in my original post. We have WinZip(R) Command Line Support Add-On Version 4.0 64-bit (Build 10562) Copyright (c) 1991-2013 WinZip International LLC. The executable is WinZp.exe.

As for WinZp.exe, it runs fine with the exact same files -- so long as I run not as part of a FILENAME PIPE.

There's something about running as part of a FILENAME PIPE that is causing things to hang.

Jim

Re: Filename Pipe X Command -- Limitations?

Patrick — Tue, 03 Mar 2020 01:43:55 GMT

Timeout is just a guess. Googling I haven't really found much in this direction except for SAS9.1.3

But.... If using a recent SAS version I'd suggest to not use a pipe at all but the filename ZIP engine.

@ChrisHemedinger wrote here quite a few blogs about it.

Re: Filename Pipe X Command -- Limitations?

Reeza — Tue, 03 Mar 2020 02:23:56 GMT

Just some quick confirmations:

1. Does the command run in the command line, not via pipe?

2. Does the command run, via WinZip manually - You've confirmed this works fine.

3. Is there enough space for the file to unzip or is there any possibility you're running out of space somehow?

4. How are you capturing results to ensure there's no error being passed back, you didn't include that portion of your code.

Is it different than what you get from #1?

Re: Filename Pipe X Command -- Limitations?

Tom — Tue, 03 Mar 2020 03:41:44 GMT

Make sure that WINZIP is not trying to ask you a question.

Also where is WINZIP writing the ZIP file? There a probably options you can set to tell it where to write the temporary file(s) it needs to complete the action. Most likely the default location is different when being called by your SAS process.

Re: Filename Pipe X Command -- Limitations?

ChrisNZ — Tue, 03 Mar 2020 03:51:13 GMT

We don't have your code but I assume it's something along the lines of:

data _null_;
  infile WinOpSys;
  input;
  putlog _INFILE_; 
run;

Is that correct?

I don't see why this step should ever time out.

This behavior is very odd.

Any reason you have LRECL in there? Did you have issues about this?

The returned output should be pretty standard.

Lastly, my DOS commands a pretty rusty, but you could maybe try to capture stderr by using :

2>&1

Re: Filename Pipe X Command -- Limitations?

jimbarbour — Tue, 03 Mar 2020 04:32:09 GMT

@Reeza,

I've run the WnZip.exe in question three ways:

1. As an X command alone.

2. As an X command embedded in a FILENAME PIPE

3. Manually from a DOS prompt (command line).

All three work equally well unless the file is larger than about 8Gb in which case number two, above, fails to complete; my job just gets stuck and goes nowhere.

I have 3.8 Tb free on my drive. I do not suspect a disk space shortage, but it is good that you ask since that must be considered. In this case, zipping a 8 Gb file should not result in a space issue on a drive with multiple terabytes free.

When I run the WnZip.exe via an X command that is part of a FILENAME PIPE, error output (STDERR) is written automatically to the SAS log, but in this case the job neither completes nor fails, and there is no error message. The job only "hangs" and goes nowhere.

When I run WinZp.exe as an X command that is not part of a FILENAME PIPE, I have had no failures, so there is no error message.

So also is there no error message when running manually from a DOS prompt. Everything works fine.

Jim

Re: Filename Pipe X Command -- Limitations?

Tom — Tue, 03 Mar 2020 12:51:46 GMT

Do you have an admin for the WIndows Server machine that can check what is happening?
Can you switch to 7z instead? I haven't used Winzip in 10+ years.

Re: Filename Pipe X Command -- Limitations?

Tom — Tue, 03 Mar 2020 15:26:24 GMT

Your terminology is a little confusing. SAS has an X command for running operating system commands. It also has the SYSTEM command. You can also use the PIPE filename engine to run an operating system command. You use the PIPE engine either in the a FILENAME statement that defines an fileref or directly in your FILE statement.

When you use the FILENAME statement it just defines the fileref, but it does not actually try to execute the command until you try to use the file. In your case when you try to READ from the file. Normally that would be in a data step, but there are many other commands/functions in SAS that can read from a file.

You did not show in the code snippet you posted how you are trying to read the file. Are you sure that step is not getting into an infinite loop because of the different messages that WINZIP is sending for the large file?

Please clarify that you know where WINZIP is writing its TEMPORARY files for both your SAS driven calls and your interactive calls. I seem to remember that used some Windows environment variable like, %TEMP%, by default. I definitely in the past had to override that because our default setup on Windows was pointing to a disk that did not have enough space for WINZIP to make the temporary files it needed.

Re: Filename Pipe X Command -- Limitations?

jimbarbour — Tue, 03 Mar 2020 19:16:54 GMT

@Patrick,

Hmmm. That's not a bad idea, using SAS itself to process the zip files.

In this case, I'm just zipping some files, and I don't need to actually look at the contents, so I'm thinking that SAS may be a bit less efficient than just using wzzip.exe directly.

What I'm doing is backing up a directory a few days before a monthly production job will update all the files.

I have a macro, %archive_dir, that gets all the names of sas7bdat or sas7bndx files and passes them to another macro, %archive_data, that calls wzzip.exe via an "X" command to zip the files into a Zip Archive that is saved into a sub directory.

I think wzzip.exe is going to be a lot more efficient for this purpose than reading each file line by line with a SAS program and then writing that file out with the ZIP parameter on a Libname.

The suggestion however is a good one, and I have certainly used the ZIP parameter on a libname in cases where I need to actually work with the data at the record level and then zip it. And of course, generally any reference to @ChrisHemedinger 's blog is well taken. 🙂

Jim

Re: Filename Pipe X Command -- Limitations?

jimbarbour — Tue, 03 Mar 2020 19:28:35 GMT

@Tom,

That's a good point to double check that wzzip.exe isn't prompting me for something. However, when I execute the exact same commands via a stand alone X command or manually from a DOS prompt, there is no dialog, so I'm not thinking at this juncture that that is the issue. In those cases where I use the X command not stand alone but embedded in a FILENAME PIPE, I do capture both STDOUT and STDERR, so I should be able to see any dialog -- but I am not.

Wzzip.exe is writing to a sub-directory named Archive. In my macro, I call another macro, cd. "cd" stand for Change Directory, and it changes the SAS working directory via the DLGCDIR function. I believe the macro is working correctly because 1) I can see a note in the SAS log (see below) and 2) because I can see the intermediate work files as Wzzip.exe processes the sas7bdat/sas7bndx files.

NOTE: The current working directory is now "I:\Commercial\monthly_data\Cornerstone\archive\Stg".

If it were helpful, I could post a screen shot of the "hung" intermediate files. I can watch them incrementally grow until at some point they freeze and the SAS job freezes as well.

To your point, it could be that Wzzip.exe is expecting some kind of response, but given that when the same commands are executed manually via a DOS prompt or via a stand alone X command there is no dialog, I'm not thinking that's the issue. I may however try doing a redirect of STDERR to the SAS log per @ChrisNZ 's suggestion, just to double check.

Jim

Re: Filename Pipe X Command -- Limitations?

Tom — Tue, 03 Mar 2020 19:38:10 GMT

You should definitely figure out how to tell Winzip where to write its temporary files.

I sounds like you are zipping multiple files into one ZIP file. So what it most likely is doing is reading the files one by one, compressing them and then adding them to the "archive", the ZIP file. So that is probably why you see it working and making progress, then it hits some large file and it does not have enough room to create the compressed version of that file and is getting hung.

Re: Filename Pipe X Command -- Limitations?

jimbarbour — Tue, 03 Mar 2020 20:52:24 GMT

@Tom and @ChrisNZ ,

Let me post some additional code.

Here is the essential code, in my macro. I can post the entire macro if that's helpful, but a lot of the code is housekeeping.

First, there is a FILENAME PIPE with an X command embedded in it. The X command executes wzzip.exe.

FILENAME	WinOpSys	PIPE		"&ZipPgmPath. -a ""&Zip_File."" ""&Full_File.""";

&ZipPgmPath is the fully qualified zip program, wzzip.exe.

%LET	ZipPgmPath						=	E:\%STR("")Program Files%STR("")\WinZip\wzzip.exe;

&Zip_File is the fully qualified Zip Archive, i.e. this is the path and file name of the .zip file I am creating.

&Full_File is the fully qualified sas data that I want to zip into the Zip Archive.

Pretty straightforward, and when executed alone (not part of a FILENAME PIPE) or manually from a DOS prompt, works fine. It also works very well with file sizes less than 8 Gb.

Then I read the STDOUT coming back from wzzip.exe using a DATA step:

DATA	_NULL_;
	LENGTH	_Curr_DateTime	$23;
	LENGTH	_Line			$255;
	LENGTH	_RC				4.;
	LENGTH	_First_Time		3.;
	LENGTH	_Total_Bytes	3.;

	RETAIN	_First_Time		1;
	RETAIN	_Total_Bytes	0;

	IF	_First_Time										THEN
		DO;
			_First_Time					=	0;
			PUTLOG	"&Nte2  %Format_Dashes(&Width)";
*						_Curr_DateTime				=	TRANWRD(PUT(DATETIME(), E8601DT23.2), 'T', ' ');
*						_Line						=	RESOLVE(CAT('&Nte1  %Format_Line(Start Zip at ', STRIP(_Curr_DateTime), '., &Width)'));
*						PUTLOG	_Line;
		END;

	INFILE	WinOpSys	TRUNCOVER	
						END				=	_Zip_Process_Complete
						LENGTH			=	_SAS_Length
						;
	INPUT	_WinZip_Msg	$VARYING32766.		_SAS_Length;

	_Line								=	RESOLVE(CAT('&Nte2  %Format_Line(%STR(', STRIP(_WinZip_Msg), '), &Width)'));
	PUTLOG	_Line;

	IF	INDEX(_WinZip_Msg, 'Total bytes=')				THEN
		DO;
			&Cmnt	PUTLOG	"&Nte2  | 'Total bytes=' has been found";
			_Total_Bytes				=	1;
		END;

	IF	_Zip_Process_Complete							THEN
		DO;
			IF	NOT	_Total_Bytes						THEN
				DO;
					PUTLOG	"&Warn2  %Format_Dashes(&Width)";
					_Line				=	RESOLVE(CAT('&Warn1  %Format_Line(&sQuote.Total bytes=&sQuote. not returned by WinZip, &Width)'));
					PUTLOG	_Line;
					_Line				=	RESOLVE(CAT('&Warn2  %Format_Line(Results are suspect.  Check zip archive., &Width)'));
					PUTLOG	_Line;
					PUTLOG	"&Warn2  %Format_Dashes(&Width)";
					_RC					=	INPUT(SYMGET('SysRC'), 4.);
					IF	_RC				<	4			THEN
						DO;
							&Cmnt	PUTLOG	"&Nte2  Setting SYSCC and SYSRC to 4";
							CALL	SYMPUTX('SYSCC', '4', 'G');
							CALL	SYMPUTX('SYSRC', '4', 'G');
						END;
				END;
			_Curr_DateTime				=	TRANWRD(PUT(DATETIME(), E8601DT23.2), 'T', ' ');
			_Line						=	RESOLVE(CAT('&Nte2  %Format_Line(End Zip at ', STRIP(_Curr_DateTime), '., &Width)'));
			PUTLOG	_Line;

			_RC							=	INPUT(SYMGET('SysRC'), 4.);

			IF	_RC						=	0			THEN
				DO;
					PUTLOG	"&Nte2  %Format_Line(Return code from zipping &File_Name = 0., &Width)";
					PUTLOG	"&Nte2  %Format_Dashes(&Width)";
					PUTLOG	"&Nte2  ";
				END;
			ELSE
				DO;
					PUTLOG	"&Err2  %Format_Dashes(&Width)";
					_Line				=	RESOLVE(CAT('&Err1  %Format_Line(Return code from zipping &File_Name = ', STRIP(PUT(_RC, 4.)), '., &Width)'));
					PUTLOG	_Line;
					PUTLOG	"&Err2  %Format_Dashes(&Width)";
					PUTLOG	"&Err2  ";
				END;
		END;
RUN;

The use of %Format_Dashes and %Format_Line just takes whatever is coming back from wzzip.exe and formats into a "box" in the SAS log:

2 The SAS System                                                                                 11:14 Tuesday, March 3, 2020
       +-----------------------------------------------------------------------------------------------------------------+
       | Start Zip at 2020-03-03 11:01:40.25.                                                                            |
       | WinZip(R) Command Line Support Add-On Version 4.0 64-bit (Build 10562)                                          |
       | Copyright (c) 1991-2013 WinZip International LLC - All Rights Reserved                                          |
       |                                                                                                                 |
       | Adding plans.sas7bdat                                                                                           |
       | Total bytes=4653056, Compressed=381416 -> 92 percent savings.                                                   |
       |                                                                                                                 |
       | Copying Zip file "I:\Commercial\monthly_data\Cornerstone\archive\Stg\Plans_Archive_2020-03-03_1113.zip".        |
       |                                                                                                                 |
       | End Zip at 2020-03-03 11:14:20.15.                                                                              |
       | Return code from zipping Plans.sas7bdat = 0.                                                                    |
       +-----------------------------------------------------------------------------------------------------------------+

I can run the X command and create an actual physical text file with the output from wzzip.exe and then use the same DATA step to read in that physical text file, but then my "Start Zip" message is wildly inaccurate, and I suspect my check of SYSRC may not work since the data step starts after wzzip.exe is completely finished. Having the "Start Zip" message isn't essential, but I'd like to be able to catch any issues that SYSRC might reveal. We're assuming we have a reliable backup and are going to start updating files after this, so we really want to know if something goes wrong. I may be being a little paranoid here. In the data step I am checking for the text "Total bytes=" which appears to be a pretty reliable indicator that the zip has worked correctly.

Jim

Re: Filename Pipe X Command -- Limitations?

ChrisNZ — Tue, 03 Mar 2020 20:52:42 GMT

If worst comes to worst, don't use infile.

Include a redirect-to-file in the command, use call system(), and read the file in a subsequent step.

A small hassle for no good reason though.

It looks like a bug to me.

I'd write a small program that creates dummy data and zips it, and then send the whole thing to tech support so they can replicate the problem.

Re: Filename Pipe X Command -- Limitations?

jimbarbour — Tue, 03 Mar 2020 21:07:06 GMT

@ChrisNZ ,

Good suggestion.

Indeed, redirecting to a physical file is my workaround for now.

*			FILENAME	WinOpSys	PIPE		"&ZipPgmPath. -a ""&Zip_File."" ""&Full_File.""";

			%cd(&Zip_Path);
			X	"del WinZip_Messages_Temp.txt & &ZipPgmPath. -a ""&Zip_File."" ""&Full_File.""	>	WinZip_Messages_Temp.txt";
			FILENAME	WinOpSys	"WinZip_Messages_Temp.txt";

I commented out my original FILENAME with the embedded PIPE command and replaced it with:

1. A call to my change directory macro (%cd) that sets the SAS working directory to the same directory as will hold the Zip Archive.

2. A stand alone X command with a redirect to a physical text file. Note that except for the redirect related code, the X command is unchanged from the original PIPE version.

3. A FILENAME statement with the same name as the original filename.

I then use the same DATA step as before except that I comment out the write of the "Start Zip" to the log since it is completely inaccurate with this set up.

My fear in this context is that I'm really not going to catch errors with my SYMGET('SYSRC'), but hopefully my verifying the presence of "Total bytes=" in the STDOUT is sufficient -- but I'm not really sure. I have seen wzzip.exe have the occasional error, but I can't reproduce such errors on demand to see if my logic as currently structured will really detect anything.

Jim

Re: Filename Pipe X Command -- Limitations?

ChrisNZ — Tue, 03 Mar 2020 21:15:55 GMT

Just an observation (probably unrelated):

I would put the quotes around the whole path. I haven't seen the syntax you use before.

Also because there are two quotes, you don't need %str.

%let ZipPgmPath	= "E:\Program Files\WinZip\wzzip.exe";

Also, what happens if you use the data step example I provided?

Just trying to isolate the root cause here. Is it the INFILE's inner workings or is it your data step's logic, even though it simple enough.

Re: Filename Pipe X Command -- Limitations?

ChrisNZ — Tue, 03 Mar 2020 21:27:59 GMT

1. Unsure if call system() sets SYSRC. Maybe worth a try(sorry can't test atm)?

2. You could add 2>&1 at the end of the command to catch STDERR

X "del WinZip_Messages_Temp.txt & &ZipPgmPath. -a ""&Zip_File."" ""&Full_File."" > WinZip_Messages_Temp.txt 2>&1";

3. My fear in this context is that I'm really not going to catch errors

- To really ensure all the zipping worked, you could open the zip file to validate it.

- Note that unless you have a file system that addresses bitrot, your zip file is never perfectly safe.

btrfs and ZFS are the only two such file systems afaik.

Re: Filename Pipe X Command -- Limitations?

Tom — Tue, 03 Mar 2020 22:45:45 GMT

I would eliminate all of that complexity. It will let you test more easily. It will also make your program easier to use.

Don't put the quotes into your macro variables, add them when you need them.

Instead of using CD just specify the fully qualified name of the ZIP file.

Instead of trying to re-direct the messages to a text file just let them come to your SAS program.

Try setting the TEMP environment variable to point to a place where you know there is enough space. Perhaps the same place as you are writing the files. I could not find real documentation on-line but this screen shot does show both TEMP and TMP environment variables. https://support.winzip.com/hc/en-us/articles/115011955108-The-PATH-environment-variable You might try setting both.

%let zippgmpath = e:\program files\winzip\wzzip.exe;
%let free_space = d:\my_big_disk\temp ;
%let zip_path = c:\my_big_disk\archives;
%let zipfile = archive_2020_03_03.zip;
%let full_file = "c:\my_other_file\*.sas7bdat" ;

data _null_;
  infile %sysfunc(quote(
set TEMP=&free_space
; "&zippgmpath" -a "&zip_path\&zipfile" &full_file 
)) pipe ;
  input;
  put _infile_;
run;

Re: Filename Pipe X Command -- Limitations?

jimbarbour — Wed, 04 Mar 2020 03:45:15 GMT

A few more data points:

1. I can definitely see intermediate zip archive work files. See directory excerpt, below. Notice that both of the intermediate zip archive work files shown froze at the same point, 3,448,766,542 bytes.

03/03/2020  06:11 PM     3,448,766,542 _Za47800
03/03/2020  06:45 PM     3,448,766,542 _Za75784
03/03/2020  05:41 PM     1,211,498,412 ProvAddresses_Archive_2020-03-03_1714.zip
03/03/2020  05:50 PM     3,283,109,761 ProvAddresses_Archive_2020-03-03_1742.zip
03/03/2020  06:21 PM     2,449,062,651 Rx_Archive_2020-03-03_1759.zip

2. Here is a second directory excerpt. I can zip with an X command embedded filename pipe (FILENAME WinOpSys PIPE "&ZipPgmPath. -a ""&Zip_File."" ""&Full_File.""";) files up to 13.9 Gb, but any of the files larger than that freeze. I said 8 to 10 Gb yesterday. I was incorrect. I can go up to about 14 Gb based on additional testing.

02/21/2020  03:12 AM    22,343,712,768 claims_allproviders.sas7bdat
02/20/2020  10:43 PM    19,576,586,240 rx.sas7bdat
02/21/2020  03:12 AM    16,598,983,680 claims_allproviders.sas7bndx
02/20/2020  10:43 PM    13,850,731,520 rx.sas7bndx
02/20/2020  09:22 PM    10,682,236,928 provaddresses.sas7bdat
02/20/2020  07:43 PM     7,105,545,216 providers_all_npitin.sas7bndx
02/20/2020  09:22 PM     6,156,108,800 provaddresses.sas7bndx

@Tom , I like that PIPE embedded within a DATA step. I haven't used that before. I suspect a PIPE used in that context will be subject to the same constraint (about 14 Gb apparently) as having a PIPE in a conventional Filename, but I'd love to be wrong. I will try it and see. Stay tuned...

@ChrisNZ , not to play favorites, but your code is a bit more different that what I have in place, so I will try your suggestion after Tom's. Thank you very much for the suggestion; it may well have a good deal of merit inasmuch as it is quite different than my original approach. And, yes, I could open the archives and see if a) the file is present and b) the file is readable, but this is part of an automated process that will archive a couple hundred files at 0030. I'm really hoping not to have to do a manual QC here. 🙂

Jim