Call macro within Set statement, read in dataset on Linux SAS 9.4

[Jumboshrimps]

Thank you for your response.

Everyone seems very pleased with this solution.

The current macro code includes:

%if %sysfunc(fileexist("&filref./&zpflname..sas7bdat.gz")) %then %do;

x "gunzip -c &zipflpth./&zpflname..sas7bdat.gz > &tmpSASwrk./&zpflname";

x "chmod 770 &tmpSASwrk./&zpflname..sas7bdat";

%end;

(testing if the data set is in .gz format, if not, then set 

the dataset as is from the directory it was found in).

Files come to us from other vendors, some in native sas7bdat format, some .gz)

Does your macro line

%let dsfilename=%scan(%scan(&filename.,-1,/),1,.).sas7bdat;

make the first line unnecessary or redundant?

Thank you again.

[Tom]

What is the purpose of the %IF statement in your current attempt at a macro?

Are you trying to only ungzip the file when the file is gzipped?   

What do you want to do if the dataset exists in normal format instead of gzipped format?

What do you want to do if both the dataset and the gzipped file exist already?

Or are you trying to check if the file exists at all. 

What do you want to do when neither version of the file exists?

[Jumboshrimps]

Excellent questions!!

Files will be in hundreds of folders througout the servers.

I was told, in no uncertain terms, there could only be ONE file in each folder (yeah, right).

Anyway taking management for their word,  the one dataset in each folder supplied by vendors will be:

Native sas7bdat.

OR

SAS dataset .gz

"What is the purpose of the %IF statement in your current attempt at a macro?"

If the file in the directory is .gz, extract to WORK temp SAS directory

else

read in the native sas7bdat file from that folder.

Are you trying to only ungzip the file when the file is gzipped?   

Yes.

What do you want to do if the dataset exists in normal format instead of gzipped format?

Simply read in the dataset with the set statement from the directory it resides in.

What do you want to do when neither version of the file exists?

I was told that could NEVER happen (yeah, ok).

Anyway, thank you so much for your responses.

[Jumboshrimps]

Upon invocation of the macro, I'm getting:

Beginning compilation of C14_UZIP_V3 using the autocall file /data/dev/utilities/sas_util/c14_uzip_v3.sas.
ERROR: Macro keyword SYSEXEC appears as text.

ERROR: A dummy macro will be compiled.

[Tom]

You are probably missing a semi-colon. 

See this example where the %SYSEXEC STATEMENT is coded in the middle of a %LET statement.

ERROR: Macro keyword SYSEXEC appears as text.
ERROR: A dummy macro will be compiled.
134   %macro xx;
135   %let x=%sysexec echo hello ;;
WARNING: Apparent invocation of macro XX not resolved.
136   %mend xx;
137   %xx;
      -
      180

ERROR 180-322: Statement is not valid or it is used out of proper order.

Also make sure you don't have unbalanced quotes (or parentheses).  You might need to restart your SAS session if you cannot figure out how to clear it.  Post your actual macro definition if you cannot find the mistake.

[Jumboshrimps]

Actual macro is below (slightly cleaned up to remove names to protect the innocent)

/*********************************************************************************************************

File name: c14_uzip_v3.sas

Macro name : c14_uzip_v3

Purpose: For a given .gz file incorporate the mechanism of unzipping the file of extension .gz
and storing in the same directory or any other directory specified in the following parameters.

Parameters:

zpflname: the file name of the zipped file.
zipflpth: path of the file in .gz file type
unzpflpth:path for the uzipped file

Sample Macro Calls:

Modifications:

*********************************************************************************************************/

options mprint mlogic symbolgen;

%macro c14_uzip_v3(zipflpth,zpflname,tmpSASwrk);

%let zipflpth = %sysget (nmepath);%put nmepath >> &zipflpth;
%let zpflname = %sysget(nmeds); %put nmeds >> &zpflname;

%put %sysfunc(getoption(work));

%let tmpSASwrk = %sysfunc(getoption(work));
%put &tmpSASwrk;

%let filref=&zipflpth.;

%if %sysfunc(fileexist("&filref./&zpflname..sas7bdat.gz")) %then %do;

%sysexec gunzip -c &zipflpth./&zpflname..sas7bdat.gz > &tmpSASwrk./&zpflname";

x "chmod 770 &tmpSASwrk./&zpflname..sas7bdat";

x "chgrp dec-pearsis &tmpSASwrk./&zpflname..sas7bdat";

%end;

[Kurt_Bremser]

Do NOT use ANY x statements in the macro. ALL external commands need to be called by macro methods, so use %sysexec.

[Tom]

Did you not understand the discussion about the impact of generating actual SAS commands in a macro that is intended to be used to generate PART of a single statement?

You do not not want to generate those X statements.

In fact you do not need those two X statements at all, so just remove them.

The first is changing the permissions on the file, but that is not needed as the permission will be set based on the piping of the output of the gzip command, not based on any permissions on the original input file, so there is no need to change them.  Similarly the group will be set using the default group of the current process that is running SAS, so there is no need to modify it.

[Tom]

Your macro logic doesn't make any sense to me.

What are environment variables nmepath and nmeds?

Why are you using their values to overwrite the values of zipflpth and zpflname that the caller of the macro has provided to the macro?

Why are you overwriting the value of tmpSASwrk that the caller of the macro has provided with the value of the SAS option WORK? (and why are you using the SAS option WORK instead of the SAS libref WORK?)

Why do you have input parameters is you are going to ignore them.

[ScottBass]

---

@Kurt_Bremser wrote:

Hi Scott, it is possible:

/* create data for testing */

libname test '$HOME/test';

data test.class;
set sashelp.class;
run;

filename oscmd pipe "gzip $HOME/test/class.sas7bdat";

data _null_;
infile oscmd;
input;
put _infile_;
run;

/* the macro with %sysexec */

%macro gzip_on_the_fly(filename,libname);
%local targetpath dsfilename;
%let targetpath=%sysfunc(pathname(&libname.));
%let dsfilename=%scan(%scan(&filename.,-1,/),1,.).sas7bdat;
%sysexec gzip -dc &filename. > &targetpath./&dsfilename.;
&libname..%scan(&dsfilename.,1,.)
%mend;

/* now use it */
data test.class1;
set
  %gzip_on_the_fly($HOME/test/class.sas7bdat.gz,work)
;
run;

The "problem" I see here is that the %sysexec is about as communicative as the X statement (read: not) and does not return any information about the success or non-success of the external command (apart from setting &SYSRC). The filename pipe I used for the compression is much better in that regard.

The macro assumes that any input file ends on

.sas7bdat.gz

Edit: included libname in the code created by the macro.

---

Hi Kurt,

Yes of course you're right :).  I use function style ("all macro statements") macros all the time, and wasn't thinking of %sysexec.

I still think this isn't the best code design - embedding the unzip within a set statement - but obviously the OP knows more about his setup than I/we do.

[Kurt_Bremser]

I would also rather have a macro that does the unzip in a controllable form (filename pipe) and lets me stop the process when the file is missing or un-decompressable (damaged).

In fact, I do have a similar macro on production that retrieves files from different types of "mainframes" and sets a libref to the converted and downloaded flat file.

[Jumboshrimps]

Scott,

Thank you for your time on this:

Everything you stated is true, true, true, and more true.

I presented your solution (call the macro, then the SET statement)

in my original User Story and got shot down.

I cannot re-design the approach.

Has to be "SET %macro(&lib.&dataset);"

No other changes are permitted.

I cannot unpack  the data to the SAS work directory before reading in the data.

Solution for this User Story has to be "in-stream" (what ever that means).

.

Thank you again.

[ScottBass]

---

@Jumboshrimps wrote:

Scott,

Thank you for your time on this:

Everything you stated is true, true, true, and more true.

I presented your solution (call the macro, then the SET statement)

in my original User Story and got shot down.

I cannot re-design the approach.

Has to be "SET %macro(&lib.&dataset);"

No other changes are permitted.

I cannot unpack  the data to the SAS work directory before reading in the data.

Solution for this User Story has to be "in-stream" (what ever that means).

.

Thank you again.

---

Classic management dictating technical decisions.

[Tom]

So let's spell out the requirements for this macro.

Input is a dataset specification.  So a either a two level name or a one level name.  For simplicity let's also assume that there is no USER libref defined nor any use of the USER system option so we can assume one level names reference WORK datasets.

Can we modify the files in the source library?  So if they specify X.Y and X point to folder /a/b/c can we modify the files in that folder?  Looks like you do NOT want to do that.  Instead you appear to be copying the unzipped file to the WORK folder.  Do you always want to use WORK? Or would it be better to have some other location where these unzipped files are written?  Perhaps we can make the macro flexible and pass in the target location as an optional parameter.

So let's take the solution from @Kurt_Bremser and flesh it out a little.

%macro gunzip(dsname,target=work);
%local libref memname targetname filename return;

%* Split DSNAME into LIBREF and MEMNAME ;
%let memname=%scan(&dsname,-1,.);
%let libref=%scan(work.&dsname,-2,.);

%* Generate name of possible GZIP file based on LIBREF and MEMNAME ;
%let filename=%sysfunc(pathname(&libref))/%sysfunc(lowcase(&memname)).sas7bdat.gz;

%if %sysfunc(exist(&dsname)) %then %let return=&dsname ;
%else %if %sysfunc(fileexist(&filename)) %then %do;
  %let targetname=%sysfunc(pathname(&target))/%sysfunc(lowcase(&memname).sas7bdat;
  %sysexec gzip -dc "&filename" > "&targetname" ;
  %let return=&target..&memname;
%end;
%else %put ERROR: Could not find &=dsname or &=filename ;
&return.
%mend;

[Jumboshrimps]

Most excellent!!!

My original solution was to simply gunzip the dataset in a temporary "gunzip"  folder, (if it was .gz, else, simply read in the 

native sas7bdat dataset from that folder), then delete the unzipped dataset (and the folder) upon completion..

Unfortunately, the target directory must be that narly SAS temp WORK directory. I was informed dozens of

programs will be accessing this macro from the SAS_macro directory, and datasets unpacked to a 

directory created by the macro (even a temporary one) could be overwritten as

the programs can run for several days.

The datasets supplied by a few dozen vendors CANNOT BE MODIFIED, MOVED in any way from the folders

they reside in.

Hope this helps. 

In the meantime I will try solution from Kurt.

Thank you.