SAS Data Integration Studio, DataFlux Data Management Studio, SAS/ACCESS, SAS Data Loader for Hadoop and others

Reading in multiple text files in DI Studio

Reply
N/A
Posts: 1

Reading in multiple text files in DI Studio


I would like to use all of the text files from a directory as multiple sources for a SAS DI job.  I know how to do this in code using a filename statement piping out to a directory, creating a SAS dataset from this file and using a macro (excuse the syntax):

data dirlist;

infile "pipedfile"

run;

%macro readwrite(filename)

data dynamicsasname;

infile "filename";

run;

%mend;

data _null_:

set dirlist;

call execute(%readwrite(filename))

run;

How can I do this in DI using the GUI interface?

Super User
Posts: 5,257

Re: Reading in multiple text files in DI Studio

First create a job thar reads the input file, use a parameter for the filename.

Then build an "outer job" with the Loop transform, and using your "pipedfile"/dirlist as input.

Put you first job within the Loop, and map your input dirlist to the job parameter.

This will make the inner job run as many times as there are records in the parameter table, and your parameter will resolve as different macro variable values for each iteration.

See the online doc/help, there are some step-by-step guides on how to build this...

Data never sleeps
Frequent Contributor
Posts: 85

Re: Reading in multiple text files in DI Studio

Yes .. Even I have used the same technique suggested by Linus in my previous organization. It always works

Respected Advisor
Posts: 3,896

Re: Reading in multiple text files in DI Studio

Is this Windows or Unix?

If there is some naming pattern to your files then you could use a wildcard in the file name of the external file, eg: ....\folder\myfiles_*.dat

This will translate into SAS code like: infile "....\folder\myfiles_*.dat" which is valid syntax and will read all files matching the pattern.

Ask a Question
Discussion stats
  • 3 replies
  • 1091 views
  • 0 likes
  • 4 in conversation