BookmarkSubscribeRSS Feed
🔒 This topic is solved and locked. Need further help from the community? Please sign in and ask a new question.
vicky07
Quartz | Level 8

Hi,

I have a dataset of X no of observations and I want to select Y no of records randomly. I know that firstobs,obs OR ranuni can be used but wondering if there is any other better way to do it.Because firstobs,obs doesn't give true random sample also ranuni requires manual input of cuts like random < 0.2 etc.,. In the below example I want to select 5125 records randomly from dataset sample.

%Let obs = 5125;

Data abc;

Set sample(firstobs = 1 obs = &obs.);

Run;

Thanks in advance for your help!

1 ACCEPTED SOLUTION

Accepted Solutions
art297
Opal | Level 21

: I like John's approach but, from my limited tests, John's code appears to either introduce a bias of some kind or at least doesn't select the first N records based on the random seed used.

I don't recommend the 3rd approach, below, but included it to compare with the other two methods.  The 2nd and 3rd methods appear to consistently provide the same results regardless of seed or sample size specified.  The first method, though, appears to always deviate slightly:

%let ssize=10;

%let seed=5;

data bigdata;

  set sashelp.class;

  do _n_=1 to 10;

    recnum+1;

    randnum=ranuni(&seed.);

    output;

  end;

run;

data sample (drop = k);

    k = &ssize. ;                    /* specify sample size required */

    if 0 then set bigdata nobs = n ; /*  get nobs, without reading anything */

    do i = 1 to n while (k > 0) ;

      if ranuni(&seed.) < k/n then do;

        k = k-1;

        set bigdata point = i ;

        output ;

      end ;

      n=n-1 ;

    end ;

    stop ;

  run ;

proc sql OUTOBS=&ssize. ;

  create table sample2 as

    select *

      from bigdata

        order by RANUNI(&seed.)

  ;

quit;

proc sort data=bigdata

          out=sample3;

  by randnum;

run;

data sample3;

  set sample3;

  if _n_ le &ssize.;

run;

View solution in original post

9 REPLIES 9
Haikuo
Onyx | Level 15

If you have SAS/STAT,  you may find proc surveyselect coming in handy:

for your case, if non-replacement :

proc surveyselect data=sample method=srs n=5125

  out=abc;

run;

Haikuo

Astounding
PROC Star

As was suggested, if you have PROC SURVEYSELECT available, that's what it was built for.  But if not, there are a few ways to go about it (all of which require entering no more than the number 5125).  Here are a few questions that would become important:

1. Is your data set large so that speed becomes important?

2. Do you need exactly 5,125 observations, or would approximately 5,125 be acceptable?

3. Can the same observation be selected more than once?

4. What should happen if the data set contains fewer than 5,125 observations?

Even if you need exactly 5,125 randomly selected unique observations, using one pass through the data, this can be done.  The programming would involve one short but somewhat complex DATA step.

Good luck.

art297
Opal | Level 21

If you don't have sas/stat, you can also do it quite easily with proc sql.  e.g.:

proc sql OUTOBS=5125 ;

  create table abc as

    select A.*

      from sample as A

        order by RANUNI(0)

;

quit;

Tom
Super User Tom
Super User

This is a common question. The most efficient method I have seen is done using a data step and uses the POINT= option on a set statement so that only the selected observations are read from the source dataset.  For an example of this code look at this posting from John Whittington in 1998.

http://listserv.uga.edu/cgi-bin/wa?A2=ind9810A&L=sas-l&D=0&P=2569

art297
Opal | Level 21

: I like John's approach but, from my limited tests, John's code appears to either introduce a bias of some kind or at least doesn't select the first N records based on the random seed used.

I don't recommend the 3rd approach, below, but included it to compare with the other two methods.  The 2nd and 3rd methods appear to consistently provide the same results regardless of seed or sample size specified.  The first method, though, appears to always deviate slightly:

%let ssize=10;

%let seed=5;

data bigdata;

  set sashelp.class;

  do _n_=1 to 10;

    recnum+1;

    randnum=ranuni(&seed.);

    output;

  end;

run;

data sample (drop = k);

    k = &ssize. ;                    /* specify sample size required */

    if 0 then set bigdata nobs = n ; /*  get nobs, without reading anything */

    do i = 1 to n while (k > 0) ;

      if ranuni(&seed.) < k/n then do;

        k = k-1;

        set bigdata point = i ;

        output ;

      end ;

      n=n-1 ;

    end ;

    stop ;

  run ;

proc sql OUTOBS=&ssize. ;

  create table sample2 as

    select *

      from bigdata

        order by RANUNI(&seed.)

  ;

quit;

proc sort data=bigdata

          out=sample3;

  by randnum;

run;

data sample3;

  set sample3;

  if _n_ le &ssize.;

run;

Tom
Super User Tom
Super User

You will get a different list of observations using the K/N method versus the assign a random number and then sort method.  But it has been proven mathematically that each observation will have the same probability of selection. 

The big advantage is when N is large relative to K as the POINT= operation will only load the needed observations. The other methods require reading all of the observations off of the disk at least once.  I have used datasets with millions of observations that take hours to process sequentially, but I can take a sample of a couple of thousand observations in seconds.

art297
Opal | Level 21

: I ran some further tests and, while I've never seen the proof you mentioned, my own tests weren't able to identify any specific bias.  John's method definitely runs faster than any of the other methods.

vicky07
Quartz | Level 8

Thank you all for your reply. I have a 3 million dataset and  tested all the 3 methods listed by Arthur and K/N method runs much faster compared the other two.  Thanks again!

sas-innovate-2024.png

Join us for SAS Innovate April 16-19 at the Aria in Las Vegas. Bring the team and save big with our group pricing for a limited time only.

Pre-conference courses and tutorials are filling up fast and are always a sellout. Register today to reserve your seat.

 

Register now!

How to Concatenate Values

Learn how use the CAT functions in SAS to join values from multiple variables into a single value.

Find more tutorials on the SAS Users YouTube channel.

Click image to register for webinarClick image to register for webinar

Classroom Training Available!

Select SAS Training centers are offering in-person courses. View upcoming courses for:

View all other training opportunities.

Discussion stats
  • 9 replies
  • 14992 views
  • 9 likes
  • 5 in conversation