topic Re: Macro and data step combination to read multiple text files as a string variable in a SAS datase in SAS Programming

Macro and data step combination to read multiple text files as a string variable in a SAS dataset

nirupmenon — Sun, 29 Nov 2020 19:44:41 GMT

Folks,

I have multiple files in a directory containing descriptive text. I want to read each file completely into a SAS dataset as variables with the filename in one column and the entire text from each file as the second variable. There should be as many rows in the output dataset as files in the directory.

I followed Chris H.'s blog https://blogs.sas.com/content/sasdummy/2016/07/12/how-to-read-the-contents-of-a-file-into-a-sas-macro-variable/ and set the code as follows:

%macro fileread; /* run this macro for each file in the directory*/
data null;
length filecontent $1000 textcontent $32627 ;
retain filecontent '';
infile "C:\ContestTextfile\&filein..txt" dlmstr='//' flowover end=last;
%if _n_ = 1 %then input filecontent;
%else %do;
input;
%let filecontent=cats(filecontent,_infile_);
%end;
if last then call symput('textcontent', filecontent);
run;
%mend fileread;

/* the data step below uses the file list in the directory to assign each file name to a macro variable */
data htmdata.accel (keep=fname filecontent); set htmdata.file_list;
length filecontent $32627 ;
call symput('filein', trim(fname)); /* put the file name from the file list in a macro variable */
%fileread; /* run the macro for each file in the file list */
%let filecontent = textcontent;
run;

What I want is two columns in the dataset: File_Name File_Content.

However, I get multiple errors with the above code: 1) both filecontent and textcontent are uninitialized; 2) the output file htmdata.accel contains only fname which was read from the file_list dataset.

Any help would appreciated. Thanks!

Nirup

Re: Macro and data step combination to read multiple text files as a string variable in a SAS datase

PaigeMiller — Sun, 29 Nov 2020 20:22:41 GMT

If you get errors, show us the LOG. Run the program again with

options mprint;

as the first command.

We need to see the complete log of one of these steps that has an error, that's 100% of the log for one of those steps, with nothing chopped out. Format the log properly by copying it as text and pasting it into the window that appears when you click on the </> icon here. Do not provide the log any other way.

Re: Macro and data step combination to read multiple text files as a string variable in a SAS datase

nirupmenon — Sun, 29 Nov 2020 20:59:36 GMT

Hey Paige,

I appreciate your reply. Recent Google search yielded https://stackoverflow.com/questions/31848367/filevar-infile-statement-to-read-in-multiple-txt-file.

So I changed the program to read two infiles (first the list of files and second the content of each file) instead of the macro in my earlier program:

options mprint;
libname htmdata "C:\Users\Owner\OneDrive - George Mason University - O365 Production\TopCoder";
filename DataIn pipe "dir C:\ContestTextfile\*.txt /S /B"; 
data htmdata.accel ;
  infile DataIn truncover;
    length fil2read $256; 
    input fil2read;
  infile xyz filevar = fil2read dlmstr='//' flowover end=last; 
    length filecontent $32627 ; 
    retain filecontent '';
    do while (not last);
     if _n_ = 1 then input filecontent;
     else do; 
        input; 
        filecontent=cats(filecontent,_infile_);
     end;
    end;
   if last then output;
keep fil2read filecontent;
run;
run;

Good news is that the program captures the filecontent variable from all files. However, the fil2read is not getting captured in the htmdata.accel as a column. I have attached the log file as a Word document.

Thanks,

Nirup

Re: Macro and data step combination to read multiple text files as a string variable in a SAS datase

PaigeMiller — Sun, 29 Nov 2020 21:51:37 GMT

Many people including me will not open Microsoft Office documents as they can be security threats. Many people including me will not download attachments from this or other forums.

Please include follow the instructions I gave to provide the log file. No other format is acceptable.

Re: Macro and data step combination to read multiple text files as a string variable in a SAS datase

Tom — Sun, 29 Nov 2020 23:00:19 GMT

FIL2READ is not part of the output dataset because you used as the variable named in the FILEVAR= option on the INFILE statement. If you want to save the value assign it to another variable that will be kept.

Re: Macro and data step combination to read multiple text files as a string variable in a SAS datase

Kurt_Bremser — Sun, 29 Nov 2020 23:00:21 GMT

The variable named as FILEVAR is not included in the output; you need to use another variable to which you assign the value from the FILEVAR= variable.

Re: Macro and data step combination to read multiple text files as a string variable in a SAS datase

nirupmenon — Mon, 30 Nov 2020 13:11:55 GMT

Thanks, Kurt. Apologies, Paige that I could not figure out the SAS program / log combination to upload.