topic Issue importing a huge zipped text 'file.txt.gz'. Tried 'gzip -c filepath'. in SAS Programming

Issue importing a huge zipped text 'file.txt.gz'. Tried 'gzip -c filepath'.

Ram_Bravos — Thu, 13 Jul 2017 18:19:32 GMT

Could not import a huge text file zipped as 'file.txt.gz'.

Tried "filename file_ref Pipe 'gzip -c filepath';" and getting "ERROR: Invalid device type." while trying to read it using infile and input statements(Data Step).

Any help is Appreciated!

Re: Issue importing a huge zipped text 'file.txt.gz'. Tried 'gzip -c filepath'.

Kurt_Bremser — Thu, 13 Jul 2017 18:22:23 GMT

When decompressing to stdout, use

gzip -dc filename

Re: Issue importing a huge zipped text 'file.txt.gz'. Tried 'gzip -c filepath'.

Ram_Bravos — Fri, 14 Jul 2017 12:59:40 GMT

Appreciate your response!

I tried that too, still same issue.

FILENAME LGCY_GZ PIPE "gzip -dc /product/..../Infile.TXT.gz" lrecl=32767;

DATA Legacy_Test;
INFILE LGCY_GZ(OBS=5)
LRECL=32767
DLM='|'
MISSOVER
DSD;
INPUT
D_U_N_S_NUMBER : BEST9.

.....

....;

run;

I get this error:

"ERROR: Invalid device type."

Re: Issue importing a huge zipped text 'file.txt.gz'. Tried 'gzip -c filepath'.

Kurt_Bremser — Fri, 14 Jul 2017 13:09:37 GMT

Do you have XCMD enabled? Run

proc options;
run;

and search for "XCMD" in the log.

filename pipe only works when XCMD is enabled.

But you can try filename zip:

filename lgcy_gz zip "/product/....../infile.txt.gz";

Re: Issue importing a huge zipped text 'file.txt.gz'. Tried 'gzip -c filepath'.

Ram_Bravos — Fri, 14 Jul 2017 15:25:59 GMT

Ran Proc Options and i could see 'XCMD' in the log.

Tried (FILENAME LGCY_GZ ZIP "/product/sas-shared/...../file.txt.gz" lrecl=32767;) getting the error below.

ERROR: The file "/product/sas-shared/......./TWC_LEGACY1.TXT.gz" exists and is not a zip file. The output
file must be a zip file.

Re: Issue importing a huge zipped text 'file.txt.gz'. Tried 'gzip -c filepath'.

Patrick — Sat, 15 Jul 2017 03:39:39 GMT

@Ram_Bravos

Unlike in your previous post you're now trying to use the ZIP engine. For this to work you would need a ZIP and not a GZ archive.

Stick to the PIPE.

First thing to do: Use Putty or the like and make your GZIP command work without any SAS involvement. Once that works stick the command into pipe using the SAS Filename command as you've already started doing.

FILENAME LGCY_GZ PIPE "gzip -dc /product/..../Infile.TXT.gz" lrecl=32767;

Make sure you're using a length for lrecl long enough for the file you want to read (infile.txt). lrecl may be longer than 32767.

And for a first test using SAS I'd go for code as below:

filename lgcy_gz pipe "gzip -dc /product/..../infile.txt.gz" lrecl=32767;
data legacy_test;
  infile lgcy_gz;
  input;
  put _infile_;
  stop;
run;

Re: Issue importing a huge zipped text 'file.txt.gz'. Tried 'gzip -c filepath'.

Kurt_Bremser — Mon, 17 Jul 2017 06:26:12 GMT

@Ram_Bravos wrote:

Ran Proc Options and i could see 'XCMD' in the log.

Tried (FILENAME LGCY_GZ ZIP "/product/sas-shared/...../file.txt.gz" lrecl=32767;) getting the error below.

ERROR: The file "/product/sas-shared/......./TWC_LEGACY1.TXT.gz" exists and is not a zip file. The output
file must be a zip file.

So you are forced to use the external gzip.

Do this:

filename oscmd pipe "gzip -d /product/..../Infile.TXT.gz 2>&1";

data _null_;
infile oscmd;
input;
put _infile_;
run;

filename oscmd clear;

You will see all messages created by the system call in your SAS log.