topic Re: Splitting large dataset into smaller datasets while keeping observations for same ID together in SAS Programming

Splitting large dataset into smaller datasets while keeping observations for same ID together

akj — Wed, 02 Jan 2019 18:31:36 GMT

I have a large file of inpatient stays that I need to split into smaller files (approx. 20,000 observations) for operational purposes. A patient (patient_id) may have one or more observations in the data set, these observations need to be kept together in the new smaller datasets, i.e., I can't split the large dataset into the first 20000 obs and then the second 20000 and so on.

How can I make sure to keep the observations for a particular patient (patient_ID) together?

Re: Splitting large dataset into smaller datasets while keeping observations for same ID together

Kurt_Bremser — Wed, 02 Jan 2019 18:39:16 GMT

Sort by patient_id.

Run this data step:

data intermediate;
set have end=eof;
by patient_id;
retain counter 0 ds_counter 1;
counter + 1;
if first.patient_id and counter ge 20000
then do;
  ds_counter + 1;
  counter = 1;
end;
if end then call symputx('num_ds',ds_counter);
drop counter;
run;

You now have the maximum number of datasets in the macro variable (for use in a %do loop), and an indicator in every observation where it should go.