topic Check all text files have same number of fields when reading multiple datasets in using wildcard in New SAS User

Check all text files have same number of fields when reading multiple datasets in using wildcard

bh11 — Thu, 14 Feb 2019 21:03:05 GMT

Hi! I am reading in several text files using a wildcard in the infile statement and I include additional statements to store the name of the file in a separate variable. Each text file includes the variable names in the first row.

DATA work.test
     LENGTH filename txt_file_name $256;
     RETAIN txt_file_name;
     INFILE "C:\samplepath\*.txt" 
          EOV = eov
          FILENAME = filename
          DLM = ","
          MISSOVER
          DSD
          LRECL = 32767;
     INPUT@;
          IF _N_ = 1 OR EOV THEN DO;
               txt_file_name = scan(filename, -1,"\");
               EOV = 0;
               DELETE;
               END;
          ELSE
       INPUT .....

The files that I am reading in 'should' all have the same structure. Is there a way to test within this data step that the files being read in all have the same number of variables? I am trying to avoid reading them all in separately and checking that way. I was thinking that replacing the MISSOVER option with the STOPOVER option might work, but I get an error message "INPUT statement exceeded record length." Any help is greatly appreciated. Thanks!

Re: Check all text files have same number of fields when reading multiple datasets in using wildcard

SASJedi — Thu, 14 Feb 2019 21:28:43 GMT

Try something like this:

/* Make flat files to play with */
title; footnote;
ods csvall file="C:\temp\t1.txt";
proc print data=sashelp.class(where=(Sex='F')) noobs;
run;
ods csvall close;
ods csvall file="C:\temp\t2.txt";
proc print data=sashelp.class(where=(Sex='M')) noobs;
run;
ods csvall close;
data _null_;
   file "C:\temp\t3.txt" dsd ;
   set sashelp.class (obs=3);
   if _n_=1 then put '"Name","Sex","Height","Weight"';
   put Name Sex Height Weight;
run;

/* Proof of Concept */
DATA work.test;
     LENGTH filename txt_file_name _header _last_header $ 256;
     RETAIN txt_file_name _header _last_header ;
     drop _:;
     INFILE "C:\temp\t*.txt" 
          EOV = eov
          FILENAME = filename
          DLM = ","
          MISSOVER
          DSD
          LRECL = 32767;
     INPUT@;
          IF _N_ = 1 OR EOV THEN DO;
               txt_file_name = scan(filename, -1,"\");
               EOV = 0;
               _header=_infile_;
               if _last_header ne _header and _n_ ne 1 then do;
                  PUT "ERROR: File structure for " txt_file_name " has changed.";
                  PUT "ERROR- New file structure:      " _header;
                  PUT "ERROR- Previous file structure: " _last_header;
                  stop;
               end;
               _last_header=_header;
               DELETE;
          END;
          ELSE INPUT Name:$15. Sex:$1. Age Height Weight;
run;

Re: Check all text files have same number of fields when reading multiple datasets in using wildcard

bh11 — Thu, 14 Feb 2019 22:08:59 GMT

I tested this using sample datasets I created where there was a variable removed, a variable added, and a variable name changed. Once I adjusted the LENGTH statement (the _header and _last_header variables were very long for my datasets) this worked perfectly. Thanks!