Hi Good Evening
suppose i have huge data in a single dataset so i want to split that data into multiple datasets following
first dataset 20%
second dataset 50%
third dataset 30%
accordingly remaing data should be existing dataset how we can do this problem
If so then do
data have;
do x=1 to 1e7;
y=rand('uniform', 1, 100);
output;
end;
run;
data one two three;
set have nobs=nobs;
if _N_/nobs <= .2 then output one;
else if .2 < _N_/nobs <= .5 then output two;
else output three;
run;
What is the logic here? The 20% in the first data set, is that the first 20% of the data from the original data or?
If so then do
data have;
do x=1 to 1e7;
y=rand('uniform', 1, 100);
output;
end;
run;
data one two three;
set have nobs=nobs;
if _N_/nobs <= .2 then output one;
else if .2 < _N_/nobs <= .5 then output two;
else output three;
run;
If you have a realy huge file and you want to deal with part of it fast
you should consider using SPDE engine, that is
1) Allocate a SPDE engine library
libname mylib spde '/disk1/spdedata';
2) Copy or move your dataset using a datastep or proc copy to the new library.
Read more in sas documentation.
Thank You sir
for your valuable point i
Why split the original into multiple data sets and waste disk space if you can create a number of views with the FIRSTOBS= OBS= data set option valued according to the required percentage? Of course, you don't have to do it manually:
data have ;
RID + 1 ;
set sashelp.cars ;
run ;
data _null_ ;
array pct p1-p3 (20 50 30) ;
lo = 1 ;
do over pct ;
cpct + pct ;
hi = ceil (cpct * divide (n, 100)) ;
call execute (catx (" ", cats ("data v", _i_), "/", cats ("view=v", _i_), ";")) ;
call execute (catx (" ", "set have (firstobs=", lo, "obs=", hi, ") ;")) ;
call execute ("run ;") ;
lo = hi + 1 ;
end ;
stop ;
set have nobs = n ;
run ;
When this code has run, you'll see this story in the log:
1 + data v1 / view=v1 ; 2 + set have (firstobs= 1 obs= 86 ) ; 3 + run ; NOTE: DATA STEP view saved on file WORK.V1. 4 + data v2 / view=v2 ; 5 + set have (firstobs= 87 obs= 300 ) ; 6 + run ; NOTE: DATA STEP view saved on file WORK.V2. 7 + data v3 / view=v3 ; 8 + set have (firstobs= 301 obs= 428 ) ; 9 + run ; NOTE: DATA STEP view saved on file WORK.V3.
Now whenever you want to process part #1 of your original file, just read view V1, and so forth.
Alternatively, instead of creating the views, you can create 3 sets of from-to macro variables:
data _null_ ;
array pct p1-p3 (20 50 30) ;
lo = 1 ;
do over pct ;
cpct + pct ;
hi = ceil (cpct * divide (n, 100)) ;
call symputx (cats ("lo", _i_), lo) ;
call symputx (cats ("hi", _i_), hi) ;
lo = hi + 1 ;
end ;
stop ;
set have nobs = n ;
run ;
Now if you want to process the specific parts of your file you'd reference the original file respectively as:
have (firstobs = &lo1 obs = &hi1) have (firstobs = &lo2 obs = &hi2) have (firstobs = &lo3 obs = &hi3)
Kind regards
Paul D.
Are you ready for the spotlight? We're accepting content ideas for SAS Innovate 2025 to be held May 6-9 in Orlando, FL. The call is open until September 25. Read more here about why you should contribute and what is in it for you!
Learn how use the CAT functions in SAS to join values from multiple variables into a single value.
Find more tutorials on the SAS Users YouTube channel.