FILENAME ERROR WHEN DYNAMICALLY GENERATING FILENAME

[shashashay]

I am trying to dynamically generate filenames through a do loop and date manipulation.

I have multiple files placed at a path. These files have the date appended in their name. I am using a cats operator to dynamically generate the path and assign it to a fileref but it errors out. Can someone please help. 

data _null_;
  startd='10feb2019'd;
	 endd='10feb2019'd;
	 dif=intck('day',startd,endd);
	do  i=0 to dif;
		date1= intnx('day',startd,i);
		format date1 yymmddn8.;
		fname = cats('F',i);
		fpath= catx(put(date1,yymmddn8.),"gunzip -c /DATA/RISK/project/baba/black/Eccvfg_input.",".ods.gz");
		filename fname fpath;
		datname = cats('DQ_',put(date1,yymmddn8.));
		put fpath;
		end;
		run;

[Amir]

Thanks for sharing your code. Please also share the log showing the data step and any error or warning messages.

Amir.

[art297]

Don't you really want to declare ZIP as the engine and enclose the path in quotes? See, e.g., https://blogs.sas.com/content/sasdummy/2017/10/10/reading-writing-gzip-files-sas/

Art, CEO, AnalystFinder.com

[ballardw]

And show an example of what the generated file names actually should look like.

I suspect the main problem of your not-stated error is that you need to have a physical file name, not a program creating files such as GUNZIP though you might get away a with a PIPE as the device though I think that would return success/fail messages.

Or perhaps you want to write the gunzip code to a program file and then execute it? Then assign filerefs to the resulting unzipped files??

[Reeza]

You can't create a filename this way in the middle of a data step. Instead, use the filename function. 

fpath= catx(put(date1,yymmddn8.),"gunzip -c /DATA/RISK/project/baba/black/Eccvfg_input.",".ods.gz");
filename fname fpath;

Instead use:

rc = filename(fname, fpath);

If rc=0 the filename was successful.

---

@shashashay wrote:

I am trying to dynamically generate filenames through a do loop and date manipulation.

I have multiple files placed at a path. These files have the date appended in their name. I am using a cats operator to dynamically generate the path and assign it to a fileref but it errors out. Can someone please help. 

data _null_;
  startd='10feb2019'd;
	 endd='10feb2019'd;
	 dif=intck('day',startd,endd);
	do  i=0 to dif;
		date1= intnx('day',startd,i);
		format date1 yymmddn8.;
		fname = cats('F',i);
		fpath= catx(put(date1,yymmddn8.),"gunzip -c /DATA/RISK/project/baba/black/Eccvfg_input.",".ods.gz");
		filename fname fpath;
		datname = cats('DQ_',put(date1,yymmddn8.));
		put fpath;
		end;
		run;

---

[Tom]

Looks like you are just trying to generate filerefs.  You cannot dynamically generate a FILENAME statement in a data step. The FILENAME statement is a global statement not a data step statement.  You could use the FILENAME() function.

Of you could just just write the FILENAME statements into a file and then %INCLUDE the file to run the generated commands.

filename code temp;
data _null_;
  file code;
  do day='10feb2019'd to '10feb2019'd;
    fileno+1;
    put 'filename F' fileno z7. 
         ' pipe "gunzip -c ''/DATA/RISK/project/baba/black/Eccvfg_input.' 
         day yymmddn8. '.ods.gz''";'
    ;
  end;
run;
%include code / source2;

But if you just want to read the data (not generate a bunch of filerefs) then use the FILEVAR option on the INFILE (or FILE statement for writing multiple data files).

data want ;
do day='10feb2019'd to '10feb2019'd;
  format day yymmdd10.;
  length fname cmd $200;
  fname=cats('/DATA/RISK/project/baba/black/Eccvfg_input.',put(day,yymmddn8.),'.ods.gz');
  cmd=catx(' ','gunzip -c',fname);
  infile mydata pipe filevar=cmd end=eof ;
  do while(not eof);
     input .... ;
     output;
  end;
end;
run;

[shashashay]

Hello Tom,

The code works and take me 90% where I need to be. I am generating multiple filerefs so that I can use them to generate multiple sas data sets. So if the loop runs from 10feb2019 to 12feb2019, I would like to have 3 file refs and corresponding three data sets. The problem I am facing is to run the do loop outside of the data step which does not seem to be an option. Is there a way to create multiple data sets for each of the file ref created when the loop runs for multiple dates? Thanks for your help.

[Tom · Accepted solution]

Use the first option I presented.

You might find that it is easier to create a little macro that can be used to process one file and then generate the calls to the macro from the file.  So instead of generating the FILENAME statements like in that code you could generate the macro calls.  You could skip the macro and have the data step generate all of the statements needed to generate one of the datasets, but that might make the data step harder to debug.

%macro do_one_file(date,filename,dataset);
filename &filename pipe 
  "gunzip -c '/DATA/RISK/project/baba/black/Eccvfg_input.&date..ods.gz'"
;
data &dataset;
  infile &filename .... ;
  ...
run;
%mend ;

filename code temp;
data _null_;
  file code;
  do day='10feb2019'd to '10feb2019'd;
    fileno+1;
    put '%do_one_file(' day yymmddn8. 
          ',F' fileno z7. ',data' fileno z7. ');'
   ;
  end;
run;
%include code / source2;

[shashashay]

Hello Tom,

Thanks for the code. For some reason the macro is not running. Please see code below. The log shows no error just couple of warnings: 

NOTE 49-169: The meaning of an identifier after a quoted string might change in a future SAS release. Inserting white space
between a quoted string and the succeeding identifier is recommended.

WARNING: The quoted string currently being processed has become more than 262 characters long. You might have unbalanced quotation
marks.

even though there are no errors, I don't see a dataset created. 

%macro do_one_file(date,filename,dataset);
filename &filename pipe 
  "gunzip -c '/DATA/RISK/project/baba/black/Eccvfg_input.&date..ods.gz'"
;
data &dataset;
  infile &filename 
  	LRECL = 5000
		MISSOVER
		firstobs=2
	;
	INPUT
		@0001 ACCT                      $CHAR016.
		@0017 LNAME                $CHAR025.
		@0042 FNAME                $CHAR020.
		@0062 MNAME                $CHAR001.
		@0063 SUFFIX               $CHAR004.
		@3568 FICO_1 		$CHAR011.
		@3579 FICO_2 		$CHAR011.
		@3590 FICO_3 			$CHAR011.
		;/*there are other fields only showing few here*/ 
run;
%mend;

filename code temp;
data _null_;
  file code;
  do day='10feb2019'd to '10feb2019'd;
    fileno+1;
    put '%do_one_file(' day yymmddn8. 
          ',F' fileno z7. ',data' fileno z7. ');'
   ;
  end;
run;
%include code / source2; 

[Tom]

You have probably gotten unbalanced quotes. If you cannot clear them restart your SAS session and try again.

[shashashay]

I restarted and it works now. Thank you for all the help!