Every other line is skipped when creating filename variable from .txt files

[USCSS_Nostromo]

Dear SAS community,

I am interested in importing ca. 60 .txt files and creating a variable name in the data set that contains the individual imported file names. I have been successful in doing this with one exception: when I include my script for creating a variable containing the file names, every other line of data is excluded/deleted.

Can you help me please adjust my script, such that this problem no longer occurs?

I am running version SAS 9.4

Here is my script. Attached are examples of the data I wish to import.

Thank you for any help!

Sincerely,

Ian

%let pathname=/*Path;

data Test ;
infile /*'"Path\*.txt" dsd LRECL=32000 firstobs=3 ;
length var1-var7 $256;
input var1-var7;

EpochNr = input(Var3,8.);
EpochLe = input(Var4,8.);
StageSl = input(Var7,8.);

length filename fname $256 ;
infile "&pathname\*.txt" filename=fname ;
array d (1000) _temporary_ ;
length dummy $8 ;
input @;
filename=scan(fname,-2,'./\') ;

drop dummy Var1 -- Var7;
if EpochNr= ' ' then delete;
if StageSl = 255 then delete;
ID =scan(filename,1,'_','m');
Night =scan(filename,2,'_');

run;

[Astounding]

At a minimum add another @:

input @@;

Before making more substantial changes, check the results.  Make sure that the very last line of data is being properly read.  (We may need to revisit and make additional changes if that is an issue.)

[USCSS_Nostromo]

Dear Astounding,

Thank you for your reply to my question! The very last lines of the data sets are okay.

What you suggest has solved the problem, but now there are two new problems:

1.) After the first file that is imported, the first line of data in all the ensuing files is deleted, i.e., each file's data set begins with line 2 instead of line 1.

2.) I notice that file RES004 has repeated numbers in the first column, such that instead of counting upwards (1,2,3,4,etc.) as it should, it has this strange pattern (2,2,3,4,4,5,6,6,7,8,8,etc.).

I have tried adding dsd truncover LRECL32000 to the line infile "&pathname\*.txt"  but this does not solve the problem.

Do you have any idea what I could do?

Thanks for all your help!

Sincerely,

Ian

[Astounding]

Sorry, this is just too difficult to debug without seeing your real code.  The program you displayed doesn't contain anything named RES004.  And there must be some point to creating the temporary array, but the program you displayed doesn't need it.  

[Kurt_Bremser]

/* is SAS syntax for the start of a comment; everything after it up to the first */ is considered comment and discarded before code is executed.

Therefore your code won't work at all.

---

@USCSS_Nostromo wrote:

Dear SAS community,

I am interested in importing ca. 60 .txt files and creating a variable name in the data set that contains the individual imported file names. I have been successful in doing this with one exception: when I include my script for creating a variable containing the file names, every other line of data is excluded/deleted.

Can you help me please adjust my script, such that this problem no longer occurs?

I am running version SAS 9.4

Here is my script. Attached are examples of the data I wish to import.

Thank you for any help!

Sincerely,

Ian

%let pathname=/*Path;

data Test ;
infile /*'"Path\*.txt" dsd LRECL=32000 firstobs=3 ;
length var1-var7 $256;
input var1-var7;

EpochNr = input(Var3,8.);
EpochLe = input(Var4,8.);
StageSl = input(Var7,8.);

length filename fname $256 ;
infile "&pathname\*.txt" filename=fname ;
array d (1000) _temporary_ ;
length dummy $8 ;
@input @;
filename=scan(fname,-2,'./\') ;

drop dummy Var1 -- Var7;
if EpochNr= ' ' then delete;
if StageSl = 255 then delete;
ID =scan(filename,1,'_','m');
Night =scan(filename,2,'_');

run;

---

[USCSS_Nostromo]

Dear Kurt Bremser,

Thank you for your reply. Thank you for pointing out this mistake. I am aware of how to /*comment*/ in SAS. I edited my script slightly before uploading it here to exclude my computer's path and forgot to delete this. Here's a better example. Can you help at all any further with my problem?

Sincerely,

Ian

%let pathname=C:\folder;

data Test ;
infile "C:\folder\*.txt" dsd LRECL=32000 firstobs=3 ;
length var1-var7 $256;
input var1-var7;

EpochNr = input(Var3,8.);
EpochLe = input(Var4,8.);
StageSl = input(Var7,8.);

length filename fname $256 ;
infile "&pathname\*.txt" filename=fname ;
array d (1000) _temporary_ ;
length dummy $8 ;
input @@;
filename=scan(fname,-2,'./\') ;

drop dummy Var1 -- Var7;
if EpochNr= ' ' then delete;
if StageSl = 255 then delete;
ID =scan(filename,1,'_','m');
Night =scan(filename,2,'_');

run;

[Kurt_Bremser]

Could it be that you basically want this:

data want;
length filename fname $254;
infile '$HOME/sascommunity/RES*_BN.TXT' dlm=',' filename=fname;
input @;
if substr(_infile_,1,1) not in ('#','-');
_infile_ = compress(_infile_,'09'x);
input segment epoch start time duration interval event;
filename=scan(scan(fname,-1,'/'),1,'.');
run;

[USCSS_Nostromo]

Dear Kurt Bremser,

Thank you! This helps a lot! I have adapted the code slightly to address my specific path and file name lengths.

I have one problem, however. The first line of the data sets following the first data set is messed up (attached PDF photo, line 3943), such that the data do not get read in correctly. Could this have to do with the hyphen that is present in the column "interval" in the final line of the data sets?

Do you have any suggestions for solving this?

Thanks again!

-Ian

Here is my lightly adjusted script:

data want;
length filename fname $254;
infile 'P:\PTSD\4. Data\EEG\EEG text files\Ians_text_files_out\Test\RES*_BN.TXT' dlm=',' filename=fname;
input @;
if substr(_infile_,1,1) not in ('#','-');
_infile_ = compress(_infile_,'09'x);
input segment epoch start time duration interval event;
filename=scan(fname,-2,'./\') ;
ID =scan(filename,1,'_','m');
Night =scan(filename,2,'_');

run;

[Kurt_Bremser · Accepted solution]

So we need to take two precautions:

1. skip empty lines at the end of the infiles
2. correctly read the hyphens for interval in the last non-empty line of the files

data want;
length filename fname $254;
infile '$HOME/sascommunity/RES*_BN.TXT' dlm=',' filename=fname;
input @;
if lengthn(_infile_) > 0; /* #1 */
if substr(_infile_,1,1) not in ('#','-');
_infile_ = compress(_infile_,'09'x);
input segment epoch start time duration _interval $ event;
filename=scan(scan(fname,-1,'/'),1,'.');
if _interval ne '-' then interval = input(_interval,best.); /* #2 */
drop _interval;
run;

You'll see that the log is now clean; additionally consider to add a truncover option to the infile statement.

[USCSS_Nostromo]

Hello Kurt Bremser,

Thank you for taking a moment to help me with my problem! Consider it solved.

Best,

Ian