I crafted a program that takes a large file, splits it into pieces, and then runs through all the obs in the 1st piece, then all the obs in the second piece, etc, until it goes through all the pieces. Its crude, but it works except for one issue. The original large dataset has a running list_number. In my small sample dataset, it goes from 1 to 129 obs. When I split it into 6 datasets of 25 or less, that list_number remains. So piece 1 has a list number 1-25. piece 2 is 26-50, etc. Unfortunately, my loop resets the list_number to 1 as it starts with piece 2, and so on and so on. Total iterations of the loop within a loop is correct, but because the list_number in piece 2 starts with 26, and the loop is looking for list_number 1, it fails to read anything from piece 2.
So, I either need to keep the list_number counter from resetting back to 1 or, when I create the splits, somehow reset all of the list_numbers.
Attached is my code. The US13 macro will eventually be filled with a regression analysis. It just beeps for now while I am debugging.
DM 'CLEAR LOG; CLEAR OUTPUT'; RESETLINE;
options ExtendObsCounter=no;
options notes source source2 mprint mlogic symbolgen error=20;
LIBNAME COMPANY 'E:\SAS Data\Regression';
/*This macro splits a data set into data sets of size N.
The parameters required are:
1. DSN = input data set name, such as sashelp.cars.
The libname should be included unless the data set
is in the work library.
2. Size = Number of records to be included in each data
set. Note that the last data set will be truncated,
ie if only 28 records are available only 28 will be
written to the output data set.
3. outDsnPrefix = Name of output data sets, will be indexed as
outDSNPrefix1
outDSNPrefix2
outDSNPrefix3
*/
%macro split (dsn=, size=, outDsnPrefix=);
%*Get number of records and calculate the number of files needed;
data _null_;
set &dsn. nobs=_nobs;
call symputx('nrecs', _nobs);
n_files=ceil(_nobs/ &size.);
call symputx('nfiles', n_files);
stop;
run;
%*Set the start and end of data set to get first data set;
%let first=1;
%let last=&size.;
%*Loop to split files;
%do i=1 %to &nfiles;
%*Split file by number of records;
data &outDsnPrefix.&i.;
set &dsn. (firstobs=&first obs=&last);
run;
%*Increment counters to have correct first/last;
%let first = %eval(&last+1);
%let last = %eval((&i. + 1)*&size.);
%end;
%mend split;
*Example call;
*After running this, you should find 9 data sets named Split1-Split9;
%split(dsn=company.testcomb130, size=25, outDsnPrefix=TESTCOMB);
%MACRO US13;
data _null_;
call sound(440,200);
run;
%MEND US13;
%macro test;
proc sql noprint;
select count(*) into :n
from testcomb&padded_number
;
quit;
%do i=1 %to &n;
data _null_;
set testcomb&padded_number;
where list_number = &i ;
call symputx('changelist',combos2);
run;
%US13;
%end;
%mend;
%macro join;
data testcomb;
set company.testcomb130;
run;
proc sql noprint;
select count(*) into :n
from testcomB
;
quit;
data _null_;
set company.testcomb130 nobs=_nobs;
call symputx('nrecs', _nobs);
n_files=ceil(_nobs/ 25);
call symputx('nfiles', n_files);
stop;
run;
%let num_files = &nfiles;
%do z=1 %to &num_files;
%let padded_number = %sysfunc(putn(&z, 4.0));
%test
%end;
%mend join;
%join
... View more
