I would not use firstobs/obs to divide the join into subgroup joins, because a given CLM_ID may be in more than one of those subgroup joins. Instead, examine each CLM_ID once, by choosing a restricted range of CLM_ID in both datasets, for each subgroup join. This can work because CLM_ID is the join variable.
Let's say you divide your CLM_ID values into 5 ranges, each range with a lower limit (LLLLLLL) and upper limit (UUUUUUU), where LLLLLLL and UUUUUUU are quintile values. Of course, the lowest range doesn't need a specified LLLLLLL, and the highest range doesn't need a specified UUUUUUU.
Then you could run five programs, such as the below - just put in values in place of LLLLLLL and RRRRRRR:
data want1;
set bigdataset (keep= list of variables);
where LLLLLLL <= clm_id < UUUUUUU;
if _n_=1 then do;
declare hash h (dataset:'header (where=(where LLLLLLL <= clm_id < UUUUUUU))');
h.definekey('clm_id');
h.definedata('clm_id');
h.definedone();
end;
if h.check()=0;
run;
Notes:
- Limiting the range in the hash object this way allows you to avoid requiring more memory than available.
- Using the "where" statement after the SET outsources the filtering of the big data set to the data set engine, saves a lot of resources.
- Then it's just a matter of seeing if the filtered CLM_ID from the big data set is also found in the hash object.
- I've coded the above ("<=" for lower limit, and "<" for upper limit) to avoid double inclusion of the quintile values. So either drop the upper limit for the highest range, or change "<" to "<=".
Of course, this requires generating the quintile CLM_ID values. You could do something like this to find the quintiles:
proc sort data=header out=header_sorted nodup;
by clm_id;
run;
data limits (drop=CLM_ID);
set header_sorted nobs=nclm;
retain quintile 1;
if _n_=1 then LLLLLLL=clm_id;
retain LLLLLLL;
if _N_ = ceil(nclm*(quintile/5));
UUUUUUU=clm_id;
output;
quintile+1;
LLLLLLL=UUUUUUU;
run;
proc print;
run;
