BookmarkSubscribeRSS Feed
Rasheed
Calcite | Level 5

I am working on SAS program where I needed to create many data sets and I have successfully created data sets from CD_1 to CD_1001000 by using following sas command

CD='CD_1':'CD_1001000';                                                                               

do s=1 to 100;                                                                               

do k = 1 to 1000;                                                                               

sk=catt(s,k);                                                                               

skn=inputn(sk,'6.',6,0);                                                                               

call execute('create ' + CD[skn] +'  from ' + C[skn] + ';');                                                                               

call execute('append from ' + C[skn] + ';') ;                                                                               

call execute('close ' + CD[skn] + ';' ); 

After doing this when want to combine all these data sets I got out of memory error in log window, I am using following command to combine data sets

Data CCD;                                                                               

set CD_:;                                                                               

rename COL1=C;                                                                               

run; 

Please help

8 REPLIES 8
LinusH
Tourmaline | Level 20

Posting the vital part of the log would help.

Also, what are you trying to achieve?

Sounds tedious to create all those data sets, and the combine them again. Perhaps there's another less resource intense way to do this.

Data never sleeps
Rasheed
Calcite | Level 5

Actually It is a long SAS program that I am working on it, in short, I am doing 100 simulations and for each simulated data set I did 1000 bootstrap sampling (from higher order cross over designs for multivariate bioequivalence) there for each combination say simulation 1 and bootstrap sample 1 I need to create a data set I called it CD_1 therefore I needed to create data set CD_1 to CD_1001000.

Log error

497  Data CCD;

498  set CD_:;

ERROR: Out of memory.

ERROR: Out of memory.

ERROR: Out of memory.

ERROR: Out of memory.

SASKiwi
PROC Star

It appears your program is producing hundreds of datasets then trying to read them all in one DATA step. I'm not surprised you are running out of memory.

I suggest you consider a different approach that doesn't involve hundred's of datasets. I've never struck a problem that requires so many.

Unfortunately I've no idea what you are trying to do with your code as you haven't described your problem so I can't suggest a better way.

Rasheed
Calcite | Level 5

Actually It is a long SAS program that I am working on it, in short, I am doing 100 simulations and for each simulated data set I did 1000 bootstrap sampling (from higher order cross over designs for multivariate bioequivalence) there for each combination say simulation 1 and bootstrap sample 1 I need to create a data set I called it CD_1 therefore I needed to create data set CD_1 to CD_1001000.

Log error

497  Data CCD;

498  set CD_:;

ERROR: Out of memory.

ERROR: Out of memory.

ERROR: Out of memory.

ERROR: Out of memory.

RW9
Diamond | Level 26 RW9
Diamond | Level 26

I would suggest you post, in the statistical forum area, what you are attempting to do.  The people there may be able to help you define a proper model for the task.  Alternatively, post your code here, some sample data and an explanation of what you are trying to achieve.  Creating that number of datasets is obviously not the way to proceed.

Ksharp
Super User

Try proc append .

Kurt_Bremser
Super User

You're trying to feed 100.000 operations into the input buffer of the main interpreter, which causes an overflow.

Rethink your basic design, and make it less complicated.

Astounding
PROC Star

Also note, are you creating unique data sets?  Can your logic distinguish s=1 and k=11 from s=11 and k=1?  You might need to use something like sk=catt(put(s,z3.), put(k,z4.));

sas-innovate-2024.png

Join us for SAS Innovate April 16-19 at the Aria in Las Vegas. Bring the team and save big with our group pricing for a limited time only.

Pre-conference courses and tutorials are filling up fast and are always a sellout. Register today to reserve your seat.

 

Register now!

How to connect to databases in SAS Viya

Need to connect to databases in SAS Viya? SAS’ David Ghan shows you two methods – via SAS/ACCESS LIBNAME and SAS Data Connector SASLIBS – in this video.

Find more tutorials on the SAS Users YouTube channel.

Discussion stats
  • 8 replies
  • 2310 views
  • 0 likes
  • 7 in conversation