topic Re: Subsample a data set with given mean and standard deviation in Statistical Procedures

Subsample a data set with given mean and standard deviation

BigD — Wed, 17 Apr 2013 16:15:22 GMT

Hi,

I have a very large dataset of clients with their ages and other attributes. Would it be possible to extract a set of records from this larger dataset such that the subset had a mean of say 55 years of age and a standard deviation of 2?

Thanks,

Bruce

Re: Subsample a data set with given mean and standard deviation

ballardw — Wed, 17 Apr 2013 17:35:27 GMT

If I had to do this I would be tempted to use proc survey select with age as the stratum variable. Put different sizes for each strata (age) such that the result would have your desired mean and deviation with a total size that you want.

I see a certain amount of trial and error as even for a given total number of records there many ways to meet your requirement.

You might simplify code by restricting ages first to something like 45 to 65 to reduce the number of strata to use.

Hint if you use this method you probably should set a seed value so you can duplicate the data later if needed.

I would start with a trial data set of age and a weight variable and run them through proc means to get an idea of the numbers I want as the strata counts for the data shape and range I want to use later.

Re: Subsample a data set with given mean and standard deviation

Rick_SAS — Wed, 17 Apr 2013 18:18:26 GMT

I agree with @Ballardw. You can use the PDF function to determine what percentage to extract for each age stratum. For details and an example, see Construct normal data from summary statistics - The DO Loop

Re: Subsample a data set with given mean and standard deviation

ballardw — Wed, 17 Apr 2013 19:41:36 GMT

After lunch and improved blood sugar I remembered we could generate a normal as specified. This will generate a data set to use with Proc SurveySelect as the SampSize option. Pick i in the do loop to match your desired size AND verify that the raw data set has at least as many persons with age for each as the _nsize_ values. This is intended to work with SRS sampling without replacement. Or you can generate sampling fractions.

data dist;
   do i = 1 to 5000;
      x = round((rand('NORMAL',55,2)),1);
      output;
   end;
   drop i;
run;

proc freq data=junk noprint;
table x / out=SampleSize (rename=(count=_nsize_) drop=percent);
run;

Re: Subsample a data set with given mean and standard deviation

BigD — Wed, 17 Apr 2013 20:29:05 GMT

Thanks so much for these ideas and help. I will be trying them out!