topic Re: how to split huge data into multiple datasets in SAS Programming

how to split huge data into multiple datasets

BrahmanandaRao — Sat, 28 Dec 2019 17:26:43 GMT

Hi Good Evening

suppose i have huge data in a single dataset so i want to split that data into multiple datasets following

first dataset 20%

second dataset 50%

third dataset 30%

accordingly remaing data should be existing dataset how we can do this problem

Re: how to split huge data into multiple datasets

PeterClemmensen — Sat, 28 Dec 2019 17:35:00 GMT

What is the logic here? The 20% in the first data set, is that the first 20% of the data from the original data or?

Re: how to split huge data into multiple datasets

PeterClemmensen — Sat, 28 Dec 2019 17:45:53 GMT

If so then do

data have;
   do x=1 to 1e7;
      y=rand('uniform', 1, 100);
      output;
   end;
run;

data one two three;
   set have nobs=nobs;
   if           _N_/nobs <= .2 then output one;
   else if .2 < _N_/nobs <= .5 then output two;
   else                             output three;
run;

Re: how to split huge data into multiple datasets

Shmuel — Sat, 28 Dec 2019 21:18:15 GMT

If you have a realy huge file and you want to deal with part of it fast

you should consider using SPDE engine, that is

1) Allocate a SPDE engine library

libname mylib spde '/disk1/spdedata';

2) Copy or move your dataset using a datastep or proc copy to the new library.

Re: how to split huge data into multiple datasets

hashman — Sat, 28 Dec 2019 22:42:03 GMT

@BrahmanandaRao:

Why split the original into multiple data sets and waste disk space if you can create a number of views with the FIRSTOBS= OBS= data set option valued according to the required percentage? Of course, you don't have to do it manually:

data have  ;                                                                                                                            
  RID + 1 ;                                                                                                                             
  set sashelp.cars ;                                                                                                                    
run ;                                                                                                                                   
                                                                                                                                        
data _null_ ;                                                                                                                           
  array pct p1-p3 (20 50 30) ;                                                                                                          
  lo = 1 ;                                                                                                                              
  do over pct ;                                                                                                                         
    cpct + pct ;                                                                                                                        
    hi = ceil (cpct * divide (n, 100)) ;                                                                                                
    call execute (catx (" ", cats ("data v", _i_), "/", cats ("view=v", _i_), ";")) ;                                                   
    call execute (catx (" ", "set have (firstobs=", lo, "obs=", hi, ") ;")) ;                                                           
    call execute ("run ;") ;                                                                                                            
    lo = hi + 1 ;                                                                                                                       
  end ;                                                                                                                                 
  stop ;                                                                                                                                
  set have nobs = n ;                                                                                                                   
run ;

When this code has run, you'll see this story in the log:

1   + data v1 / view=v1 ;                                                                                                               
2   + set have (firstobs= 1 obs= 86 ) ;                                                                                                 
3   + run ;                                                                                                                             
NOTE: DATA STEP view saved on file WORK.V1.                                                                                             
                                                                                                                                        
4   + data v2 / view=v2 ;                                                                                                               
5   + set have (firstobs= 87 obs= 300 ) ;                                                                                               
6   + run ;                                                                                                                             
NOTE: DATA STEP view saved on file WORK.V2.                                                                                             
                                                                                                                                        
7   + data v3 / view=v3 ;                                                                                                               
8   + set have (firstobs= 301 obs= 428 ) ;                                                                                              
9   + run ;                                                                                                                             
NOTE: DATA STEP view saved on file WORK.V3.

Now whenever you want to process part #1 of your original file, just read view V1, and so forth.

Alternatively, instead of creating the views, you can create 3 sets of from-to macro variables:

data _null_ ;                                                                                                                           
  array pct p1-p3 (20 50 30) ;                                                                                                          
  lo = 1 ;                                                                                                                              
  do over pct ;                                                                                                                         
    cpct + pct ;                                                                                                                        
    hi = ceil (cpct * divide (n, 100)) ;                                                                                                
    call symputx (cats ("lo", _i_), lo) ;                                                                                               
    call symputx (cats ("hi", _i_), hi) ;                                                                                               
    lo = hi + 1 ;                                                                                                                       
  end ;                                                                                                                                 
  stop ;                                                                                                                                
  set have nobs = n ;                                                                                                                   
run ;

Now if you want to process the specific parts of your file you'd reference the original file respectively as:

  have (firstobs = &lo1 obs = &hi1) 
  have (firstobs = &lo2 obs = &hi2) 
  have (firstobs = &lo3 obs = &hi3)

Kind regards

Paul D.

Re: how to split huge data into multiple datasets

BrahmanandaRao — Sun, 29 Dec 2019 03:33:09 GMT

Thank You sir

for your valuable point i