topic reading files in a directory - help with space in directory name... in SAS Programming

reading files in a directory - help with space in directory name...

asishgautam — Mon, 12 Sep 2011 19:55:19 GMT

this is the code i am using to read a couple of text files but i keep getting a message telling me that "the system cannot find the path specified". from what i have gathered, i think my problem has to do with the spaces in the directory location. could anyone point me in direction to resolve this problem?

code:

filename indata pipe '"dir C:\Documents and Settings\agautam\Desktop\Selfgen\Model\Commercial\Commercial_Model_Input_Output_Data\CED_2011\Preliminary\Floorspace\*.txt" /b' ;

data Floor_sp ;

infile indata ;

input f2r :$200. ;

fil2read = 'C:\Documents and Settings\agautam\Desktop\Selfgen\Model\Commercial\Commercial_Model_Input_Output_Data\CED_2011\Preliminary\Floorspace\' || f2r ;

infile dummy filevar = fil2read end = done firstobs = 2 ;

do while (not done) ;

input @5 CZ 2.

@10 BTYPE $14.

@27 YEAR 4.

@35 HISTF 8.4

@47 BCASTF 8.4

@59 ADDITIONS 8.4 ;

output ;

end ;

run ;

Re: reading files in a directory - help with space in directory name...

data_null__ — Mon, 12 Sep 2011 20:24:12 GMT

If there is space in output from PIPE the input statement

input f2r :$200. ;

will only read to the first blank. The colon is for LIST input. Remove the colon and add TRUNCOVER to the INFILE INDATA.

reading files in a directory - help with space in directory name...

FriedEgg — Mon, 12 Sep 2011 20:30:00 GMT

The pipe is not necessary for what you are trying to accomplish.

filename indata 'C:\Documents and Settings\agautam\Desktop\Selfgen\Model\Commercial\Commercial_Model_Input_Output_Data\CED_2011\Preliminary\Floorspace\*.txt';

data Floor_sp;

infile indata filevar = fil2read end = done firstobs = 2 ;

input @5 CZ 2.

@10 BTYPE $14.

@27 YEAR 4.

@35 HISTF 8.4

@47 BCASTF 8.4

@59 ADDITIONS 8.4 ;

run ;

reading files in a directory - help with space in directory name...

asishgautam — Mon, 12 Sep 2011 20:39:00 GMT

okay this is what i tried but i keep getting the same message.

Stderr output:

The system cannot find the path specified.

NOTE: 0 records were read from the infile INDATA.

NOTE: The data set WORK.FLOOR_SP has 0 observations and 8 variables.

NOTE: DATA statement used (Total process time):

real time 0.04 seconds

cpu time 0.03 seconds

data Floor_sp ;

infile indata ;

input f2r $200. TRUNCOVER ;

fil2read = 'C:\Documents and Settings\agautam\Desktop\Selfgen\Model\Commercial\Commercial_Model_Input_Output_Data\CED_2011\Preliminary\Floorspace\' || f2r ;

infile dummy filevar = fil2read end = done firstobs = 2 ;

do while (not done) ;

input @5 CZ 2.

@10 BTYPE $14.

@27 YEAR 4.

@35 HISTF 8.4

@47 BCASTF 8.4

@59 ADDITIONS 8.4 ;

output ;

end ;

run ;

reading files in a directory - help with space in directory name...

asishgautam — Mon, 12 Sep 2011 20:46:43 GMT

i used the program you listed and saw quite a few errors..i am posting them below:

ERROR: Invalid physical name.

fil2read= done=0 CZ=. BTYPE= YEAR=. HISTF=. BCASTF=. ADDITIONS=. _ERROR_=1 _N_=1

NOTE: The SAS System stopped processing this step because of errors.

WARNING: The data set WORK.FLOOR_SP may be incomplete. When this step was stopped there were 0

observations and 6 variables.

WARNING: Data set WORK.FLOOR_SP was not replaced because this step was stopped.

NOTE: DATA statement used (Total process time):

real time 0.03 seconds

cpu time 0.00 seconds

reading files in a directory - help with space in directory name...

data_null__ — Mon, 12 Sep 2011 21:25:12 GMT

You will have to resolve the problem with the values of the variable FIL2READ. Use PUTLOG to inspect the value. Copy the value from the log and see if you can use it to open the file.

reading files in a directory - help with space in directory name...

art297 — Mon, 12 Sep 2011 21:26:11 GMT

Back to your original code, I have a suggestion, a comment and a question.

The suggestion: try it with double quotes at the start and end, and double double quotes around the file name. i.e.

filename indata pipe "dir ""C:\Documents and Settings\agautam\Desktop\Selfgen\Model\Commercial\Commercial_Model_Input_Output_Data\CED_2011\Preliminary\Floorspace\*.txt"" /b ;

The comment: you are likely to run into problems with your next use of the directory as double quotes will likely be needed.

The question: what are you trying to do? At first I thought you were trying to get a file (via the piped dir), then getting a 2nd file name from that file, then trying to read the file. Is that correct or do you just want to read the file?

reading files in a directory - help with space in directory name...

asishgautam — Mon, 12 Sep 2011 21:43:40 GMT

i have 3 text files that i am trying to read. they are identical files (in terms of layout) but differe by one category. the first input should get me a list of files and then next input section "should" read in each file.

reading files in a directory - help with space in directory name...

Tom — Mon, 12 Sep 2011 22:01:33 GMT

Take the DIR command out the double quotes you have around the filename in your DOS command.

filename indata pipe 'dir "C:\Documents and Settings\agautam\Desktop\Selfgen\Model\Commercial\Commercial_Model_Input_Output_Data\CED_2011\Preliminary\Floorspace\*.txt" /b' ;

Your string is very long maybe it is wrapping in the editor? Try breaking it into pieces using macros variables. Use dquotes around the command and double the quotes that you want to send to DOS.

%let root=C:\Documents and Settings\agautam\Desktop\Selfgen\Model\Commercial\Commercial_Model_Input_Output_Data;

%let project=CED_2011\Preliminary\Floorspace;

filename indata pipe "dir "&root\&project\*.txt"" /b" ;

TRUNCOVER is an option on the INFILE statement, not the INPUT statement.

Add the TRUNCOVER option to both of your INFILE statements.

reading files in a directory - help with space in directory name...

BobD — Mon, 12 Sep 2011 22:03:57 GMT

Yes, in your original program, move the first double-quote immediately before the file path; you have it in front of the dir command. In other words, change it to this:

filename indata pipe 'dir "C:\Documents and Settings\agautam\Desktop\Selfgen\Model\Commercial\Commercial_Model_Input_Output_Data\CED_2011\Preliminary\Floorspace\*.txt" /b' ;

Also, I suggest you use the TRUNCOVER option on your INFILE statement, and define your FIL2READ variable to a specific length (like $200).

reading files in a directory - help with space in directory name...

art297 — Mon, 12 Sep 2011 22:05:34 GMT

I actually like fried eggs' suggestion, but double quote the directory and leave off the filevar option. e.g.,

filename indata "C:\art\test folder\*.txt";

data Floor_sp;

infile indata end = done firstobs = 2 ;

input x;

run ;

reading files in a directory - help with space in directory name...

BobD — Mon, 12 Sep 2011 22:17:14 GMT

Yes, that would work, except that the FIRSTOBS option would only skip the first line of the first file. I'm assuming the OP wants to skip the header record on each file, hence using the FILEVAR makes sense.

reading files in a directory - help with space in directory name...

asishgautam — Mon, 12 Sep 2011 22:17:37 GMT

thank you so much!!!

reading files in a directory - help with space in directory name...

art297 — Mon, 12 Sep 2011 22:40:02 GMT

Glad to hear that you solved your problem, but I still think that my (extremely slight) modification of FriedEgg's code would be your easiest and least troublesome way of solving that particular problem.

And, while you apparently didn't confront the additional problem, I did, and for those who read this thread and can't figure out why a piped dir isn't working for them, they should take a look at: http://support.sas.com/kb/41/863.html

There is a problem with reading pipes on 64-bit window's machines. One that can be circumvented, but a problem nonetheless.

reading files in a directory - help with space in directory name...

FriedEgg — Mon, 12 Sep 2011 22:44:43 GMT

/* create some dummy files */

data _null_;

do i=1 to 3;

call execute('filename file' || strip(i) || ' "/home/friedegg/file' || strip(i) || '.txt";');

call execute('data _null_;');

call execute(' file file' || strip(i) || ';');

call execute(' put "a b c d e";');

call execute(' put "f g h i j";');

call execute('run;');

end;

run;

filename indata '/home/mkastin/file*.txt';

data want;

length _file _filex $512;

infile indata end=done filename=_file;

input (blah1-blah5) (:$1.);

retain _filex;

if _filex ne _file then

do;

_filex=_file;

delete;

end;

drop _:;

run;

/* remove this junk */

data _null_;

do i=1 to 3;

call execute('x "rm -f /home/friedegg/file' || strip(i) || '.txt";');

end;

run;

Re: reading files in a directory - help with space in directory name...

data_null__ — Tue, 13 Sep 2011 02:39:39 GMT

Look up EOV INFILE statement option it is designed to detect the start of a new file in a concatenation. Much easier than comparing file names.

Re: reading files in a directory - help with space in directory name...

FriedEgg — Tue, 13 Sep 2011 03:01:03 GMT

Thanks _null_, I had forgotten about that option.

/* create some dummy files */

data _null_;

do i=1 to 3;

call execute('filename file' || strip(i) || ' "/home/friedegg/file' || strip(i) || '.txt";');

call execute('data _null_;');

call execute(' file file' || strip(i) || ';');

call execute(' put "a b c d e";');

call execute(' put "f g h i j";');

call execute('run;');

end;

run;

filename indata '/home/mkastin/file*.txt';

data want;

length _file _filex $512;

infile indata eov=_firstrec_;

input (blah1-blah5) (:$1.);

if _n_ or _firstrec_ then delete; /* _firstrec_ will =1 for all files first record except the first */

run;

/* remove this junk */

data _null_;

do i=1 to 3;

call execute('x "rm -f /home/friedegg/file' || strip(i) || '.txt";');

end;

run;