input thousands of text files

Accepted Solution Solved
Reply
Contributor
Posts: 51
Accepted Solution

input thousands of text files

Hi everyone,

I have over a million of text files in a folder, I want to input all and create a new column in the combined SAS dataset with the name of text files.

Can you let me know how to do so?

Thank you so much,


Accepted Solutions
Solution
‎01-06-2014 04:20 PM
Super User
Posts: 17,899

Re: input thousands of text files

All to one dataset or each in their own dataset?

Assuming they have the same structure the following is a starting point:


data try01;

length filename txt_file_name $256;

retain txt_file_name;

infile "Path\*.txt" eov=eov filename=filename truncover;

 

if _n_ eq 1 or eov then do;

txt_file_name = scan(filename, -2, ".\");

eov=0;

end;

input

*Place input code here;

run;

View solution in original post


All Replies
Solution
‎01-06-2014 04:20 PM
Super User
Posts: 17,899

Re: input thousands of text files

All to one dataset or each in their own dataset?

Assuming they have the same structure the following is a starting point:


data try01;

length filename txt_file_name $256;

retain txt_file_name;

infile "Path\*.txt" eov=eov filename=filename truncover;

 

if _n_ eq 1 or eov then do;

txt_file_name = scan(filename, -2, ".\");

eov=0;

end;

input

*Place input code here;

run;

Super User
Super User
Posts: 6,502

Re: input thousands of text files

Key things to watch.

1) The variable specified in the FILENAME= option of the INFILE statement will be dropped. So you need to copy the value to a new variable to have it appear on the output dataset.

2) Easiest to just copy the name AFTER the INPUT statement.

length tempfilename filename $256. ;

INFILE '*' FILENAME=tempfilename .... ;

INPUT .... ;

filename = tempfilename ;

....

Contributor
Posts: 51

Re: input thousands of text files

Hi Reeza, Patrick, FriedEgg, PGStats, and Tom,

Thank you so much for your help.

Respected Advisor
Posts: 3,899

Re: input thousands of text files

Following the link to a paper which should give you all you need: http://support.sas.com/techsup/technote/ts581.pdf

Trusted Advisor
Posts: 1,300

Re: input thousands of text files

use the filename= option on the file statement in your datastep to get the names of the file.

SAS(R) 9.2 Language Reference: Dictionary, Fourth Edition

Respected Advisor
Posts: 4,654

Re: input thousands of text files

Use wildcards in your INFILE statement to read all the files at once and use option FILENAME= to create a variable that will contain the names of the files being read.

Something like:

length myFile $200;

infile "c:\myDir\*.txt" filename=myFile;

- PG

PG
☑ This topic is solved.

Need further help from the community? Please ask a new question.

Discussion stats
  • 6 replies
  • 780 views
  • 7 likes
  • 6 in conversation