topic Re: Requesting ideas to efficiently read multiple external files? in SAS Programming

Requesting ideas to efficiently read multiple external files?

novinosrin — Fri, 06 Dec 2019 18:33:53 GMT

Hello All, my objective is read a set of files(delimited) from a folder. The physical filename of the files have varying patterns and the need for me is to read certain files that follows the suit with the same pattern.

First of all, I am trying find ways of taking advantage of FILENAME statement pointing an aggregate location and read files in one shot of course with FILEVAR=. The plan seems ok but I first want to take a look at the list of filenames in the directory in programmed style unlike opening the folders manually to check.

1. is there an easy straight forward way to list all the files in the directory?

2. Once I can get 1 to work, I would read that using INPUT, write some REGEX to match the pattern and call that using FILEVAR= and be over with it.

Alternatively is there any better or efficient way to do the same that you could recommend? Please point me to the documentation if any. Thank you!

Re: Requesting ideas to efficiently read multiple external files?

Reeza — Fri, 06 Dec 2019 18:42:24 GMT

1. http://support.sas.com/kb/45/805.html

https://documentation.sas.com/?docsetId=mcrolref&docsetTarget=n0js70lrkxo6uvn1fl4a5aafnlgt.htm&docsetVersion=9.4&locale=en

2. Limit your list as you read it in to match the pattern and then pass that to the FILEVAR option.

You can do this in one or data steps really. I'd do it in two because it's easier to debug and manage that way.

Using a wildcard is also an option, I think you know this already though

https://communities.sas.com/t5/SAS-Communities-Library/How-do-I-write-a-macro-to-import-multiple-text-files-that-have/ta-p/223627

Re: Requesting ideas to efficiently read multiple external files?

tomrvincent — Fri, 06 Dec 2019 19:00:46 GMT

1. yes, depending on your O/S...which is...?

Re: Requesting ideas to efficiently read multiple external files?

novinosrin — Fri, 06 Dec 2019 19:11:34 GMT

"2. Limit your list as you read it in to match the pattern and then pass that to the FILEVAR option." -- Slick presence of mind!

I knew of the wild card, but honestly didn't cross my mind. Thank you!

@Reeza wrote:

1. http://support.sas.com/kb/45/805.html

https://documentation.sas.com/?docsetId=mcrolref&docsetTarget=n0js70lrkxo6uvn1fl4a5aafnlgt.htm&docsetVersion=9.4&locale=en

2. Limit your list as you read it in to match the pattern and then pass that to the FILEVAR option.

You can do this in one or data steps really. I'd do it in two because it's easier to debug and manage that way.

Using a wildcard is also an option, I think you know this already though

https://communities.sas.com/t5/SAS-Communities-Library/How-do-I-write-a-macro-to-import-multiple-text-files-that-have/ta-p/223627

Re: Requesting ideas to efficiently read multiple external files?

novinosrin — Fri, 06 Dec 2019 21:46:56 GMT

@Reeza Sorry but a silly question because of my inexperience. When I open a lot of text files, the header with col names looks like is split inton two lines, but firstobs=2 reads them correctly. Does that mean the values are wrapped?

Re: Requesting ideas to efficiently read multiple external files?

Reeza — Sat, 07 Dec 2019 07:16:44 GMT

Yup. Use a text editor like notepad++ to open text files. Then you can set options such as show invisible characters, such as line feeds and tabs. Makes it easier to debug things.

You can also do this in Notepad by adjusting the word wrap option.

Re: Requesting ideas to efficiently read multiple external files?

Ksharp — Sat, 07 Dec 2019 11:33:06 GMT

data file_name;
rc=filename('fid','c:\temp\');
did=dopen('fid');
do i=1 to dnum(did);
 fname='c:\temp\'||dread(did,i);	output;
end;
run;