Hi Good Evening
suppose i have huge data in a single dataset so i want to split that data into multiple datasets following
first dataset 20%
second dataset 50%
third dataset 30%
accordingly remaing data should be existing dataset how we can do this problem
If so then do
data have;
do x=1 to 1e7;
y=rand('uniform', 1, 100);
output;
end;
run;
data one two three;
set have nobs=nobs;
if _N_/nobs <= .2 then output one;
else if .2 < _N_/nobs <= .5 then output two;
else output three;
run;
What is the logic here? The 20% in the first data set, is that the first 20% of the data from the original data or?
If so then do
data have;
do x=1 to 1e7;
y=rand('uniform', 1, 100);
output;
end;
run;
data one two three;
set have nobs=nobs;
if _N_/nobs <= .2 then output one;
else if .2 < _N_/nobs <= .5 then output two;
else output three;
run;
If you have a realy huge file and you want to deal with part of it fast
you should consider using SPDE engine, that is
1) Allocate a SPDE engine library
libname mylib spde '/disk1/spdedata';
2) Copy or move your dataset using a datastep or proc copy to the new library.
Read more in sas documentation.
Thank You sir
for your valuable point i
Why split the original into multiple data sets and waste disk space if you can create a number of views with the FIRSTOBS= OBS= data set option valued according to the required percentage? Of course, you don't have to do it manually:
data have ;
RID + 1 ;
set sashelp.cars ;
run ;
data _null_ ;
array pct p1-p3 (20 50 30) ;
lo = 1 ;
do over pct ;
cpct + pct ;
hi = ceil (cpct * divide (n, 100)) ;
call execute (catx (" ", cats ("data v", _i_), "/", cats ("view=v", _i_), ";")) ;
call execute (catx (" ", "set have (firstobs=", lo, "obs=", hi, ") ;")) ;
call execute ("run ;") ;
lo = hi + 1 ;
end ;
stop ;
set have nobs = n ;
run ;
When this code has run, you'll see this story in the log:
1 + data v1 / view=v1 ; 2 + set have (firstobs= 1 obs= 86 ) ; 3 + run ; NOTE: DATA STEP view saved on file WORK.V1. 4 + data v2 / view=v2 ; 5 + set have (firstobs= 87 obs= 300 ) ; 6 + run ; NOTE: DATA STEP view saved on file WORK.V2. 7 + data v3 / view=v3 ; 8 + set have (firstobs= 301 obs= 428 ) ; 9 + run ; NOTE: DATA STEP view saved on file WORK.V3.
Now whenever you want to process part #1 of your original file, just read view V1, and so forth.
Alternatively, instead of creating the views, you can create 3 sets of from-to macro variables:
data _null_ ;
array pct p1-p3 (20 50 30) ;
lo = 1 ;
do over pct ;
cpct + pct ;
hi = ceil (cpct * divide (n, 100)) ;
call symputx (cats ("lo", _i_), lo) ;
call symputx (cats ("hi", _i_), hi) ;
lo = hi + 1 ;
end ;
stop ;
set have nobs = n ;
run ;
Now if you want to process the specific parts of your file you'd reference the original file respectively as:
have (firstobs = &lo1 obs = &hi1) have (firstobs = &lo2 obs = &hi2) have (firstobs = &lo3 obs = &hi3)
Kind regards
Paul D.
Registration is now open for SAS Innovate 2025 , our biggest and most exciting global event of the year! Join us in Orlando, FL, May 6-9.
Sign up by Dec. 31 to get the 2024 rate of just $495.
Register now!
Learn how use the CAT functions in SAS to join values from multiple variables into a single value.
Find more tutorials on the SAS Users YouTube channel.
Ready to level-up your skills? Choose your own adventure.