topic Re: trying to concatenate 400 files using macro in SAS Programming

trying to concatenate 400 files using macro

SN — Tue, 05 Mar 2013 15:19:10 GMT

I have 400 files to merge as one dataset. They have the same columns, but are of different subjects. So every line in each of the data set needs to be included.

not sure if the "merge by" would work here. Please help

Re: trying to concatenate 400 files using macro

PGStats — Tue, 05 Mar 2013 15:24:58 GMT

You might not need macro at all. If your files are text you could read them all in one datastep using wildcards in the infile statement (at least on Windows).

Re: trying to concatenate 400 files using macro

PaigeMiller — Tue, 05 Mar 2013 15:25:23 GMT

It's not clear that "MERGE" is the way to go. I think SET makes more sense.

Are these 400 files SAS data sets, or external files?

Depending on how the files are named, and what type of files you are working with, we can probably provide more specific advice.

Re: trying to concatenate 400 files using macro

SN — Tue, 05 Mar 2013 15:28:24 GMT

they are as SAS datasets in a lib ; Number of rows varies from 1 to 50 rows in the datasets. Finally, after combining the datasets, I would need to run stats analysis on it.

Re: trying to concatenate 400 files using macro

PaigeMiller — Tue, 05 Mar 2013 15:31:57 GMT

I said "depending on how the files are named"

You didn't address this. Are they named sequentially, or by some identifier, or otherwise? What are some of the names?

Re: trying to concatenate 400 files using macro

SN — Tue, 05 Mar 2013 15:35:12 GMT

Each dataset is a subject data labelled alphanumerically in a random order. There is no pattern in the dataset names.

I have a list of the 400 names in a seperate dataset. I need to write a script which would do this

data=combineddata;

set ds1...ds400;

How do I use macro to call each of the file from the filename dataset and combine them together.

Re: trying to concatenate 400 files using macro

PaigeMiller — Tue, 05 Mar 2013 15:41:30 GMT

Create a macro variable &names that contains all the data set names, replace yourlibnamegoeshere with the actual name of the library, all in UPPER CASE

proc sql noprint;

select distinct memname into :names separated by ' ' from sashelp.vtable where libname="yourlibnamegoeshere";

quit;

Then concatenate all data sets

data all;

set &names;

...

Re: trying to concatenate 400 files using macro

SN — Tue, 05 Mar 2013 16:33:06 GMT

Thanks much. this should work.

But I am currently getting an error for most of the variables

ERROR: Variable SBP has been defined as both character and numeric.

ERROR: Variable DBP has been defined as both character and numeric.

Probably the way forward is to check all the files for the variable type and run this code again.

Re: trying to concatenate 400 files using macro

SN — Thu, 07 Mar 2013 16:41:33 GMT

Thanks much. The code worked. I found a couple of empty files which gave an error as above.

Re: trying to concatenate 400 files using macro

data_null__ — Fri, 08 Mar 2013 15:31:51 GMT

With 400 datasets on the SET statement you may want to look at the SET statement documentation concerning the OPEN=DEFER option.