topic Re: split dataset into n folds in SAS Programming

split dataset into n folds

csetzkorn — Tue, 12 Dec 2017 15:58:41 GMT

I would like to split a given dataset into n stratified equal sized-ish folds by amending it with an additional column containing n. What is a common/simple way to achieve this? Thanks.

Re: split dataset into n folds

Reeza — Tue, 12 Dec 2017 16:09:29 GMT

Generally, it's not a good idea.

That being said, here's two write ups on it.

1. http://www.sascommunity.org/wiki/Split_Data_into_Subsets

2. https://blogs.sas.com/content/sasdummy/2015/01/26/how-to-split-one-data-set-into-many/

Re: split dataset into n folds

csetzkorn — Tue, 12 Dec 2017 16:28:55 GMT

I came across these references when I googled. They do not seem to split the original dataset randomly but rather based of column values. I would like to split randomly never mind stratification.

Re: split dataset into n folds

Reeza — Tue, 12 Dec 2017 16:30:35 GMT

PROC SURVEYSELECT then? Choose N samples of X data?

Re: split dataset into n folds

csetzkorn — Tue, 12 Dec 2017 16:50:47 GMT

Thanks. I thought about using PROC SURVEYSELECT. Can you please make an example for 3 folds? source dataset: Have, output datasets: wants1, wants2, wants3? Thanks

Re: split dataset into n folds

Reeza — Tue, 12 Dec 2017 16:57:41 GMT

It won't create multiple data sets but will do the random selection. Then you can use the methods above to split.

Or the manual way of adding a random numbers, sort by random number and use any of the methods in the link above.

Re: split dataset into n folds

ballardw — Tue, 12 Dec 2017 17:24:13 GMT

No guarantee about randomish result but

data want;

set have;

split = mod(_n_, 9);

run;

will add a variable that will split the data set into in 9 parts and the size difference will be plus/minus 1 between any groups. Replace 9 with your desired number.

If randomization critical then add a variable the result of a random number function, sort by that variable and then use the method above.

Re: split dataset into n folds

csetzkorn — Tue, 12 Dec 2017 17:32:04 GMT

That's fair enough but can you please show code that creates computed column with 1, 2 and 3 in it to indicate fold (see original question).

Re: split dataset into n folds

Reeza — Tue, 12 Dec 2017 17:32:49 GMT

You should have enough information and samples here to write the sample code yourself or at minimum provide sample data

Re: split dataset into n folds

csetzkorn — Tue, 12 Dec 2017 17:32:49 GMT

Thanks randomness is important

Re: split dataset into n folds

Ksharp — Wed, 13 Dec 2017 13:17:48 GMT

data have;
 set sashelp.heart;
 call streaminit(12345678);
 random=rand('uniform');
run;

proc rank data=have out=temp groups=3;
var random;
ranks group;
run;
data want1 want2 want3;
 set temp;
 if group=0 then output want1;
  else  if group=1 then output want2;
   else output want3;
run;

Re: split dataset into n folds

mkeintz — Wed, 13 Dec 2017 19:25:57 GMT

Using call rantbl, with regular updating of table probabilities will allow a single-step solution, creating the new variable subgroup.

data want (drop=_:);
  set have nobs=nrecs;

  array needed{10} _temporary_;
  array needprob{10} _temporary_;

  if _n_=1 then do;
    do _I=1 to dim(needed); 
      needed{_I}=floor(nrecs/dim(needed));
    end;
    do _I=1 to dim(needed) while (sum(of needed{*})<nrecs);
      needed{_I}=needed{_I}+1;
    end;
  end;

  _nleft = nrecs-(_n_-1);
  do _I=1 to dim(needed);
    needprob{_I}=needed{_I}/_nleft;
  end;

  seed=1250666;
  call rantbl(_seed,of needprob{*},subgroup);

  needed{subgroup}=needed{subgroup}-1;
run;

Notes:

Changing the dimension of arrays NEEDED and NEEDPROB is all that's required to change the number of randomly populated subgroups.
NEEDED tracks, for each subgroup, the number of observations yet to be added. It's dynamically updated with every incoming observations. The minimum and maximum starting values for NEEDED will differ by no more than one, and will start out summing to NRECS.
NEEDPROB array is required by the CALL RANTBL routine. It uses elements of NEEDED divided by the number of observations remaining to be assigned.