SAS Data Integration Studio, DataFlux Data Management Studio, SAS/ACCESS, SAS Data Loader for Hadoop and others

Out of Memory Error

Reply
Frequent Contributor
Posts: 84

Out of Memory Error

I am working on SAS program where I needed to create many data sets and I have successfully created data sets from CD_1 to CD_1001000 by using following sas command

CD='CD_1':'CD_1001000';                                                                               

do s=1 to 100;                                                                               

do k = 1 to 1000;                                                                               

sk=catt(s,k);                                                                               

skn=inputn(sk,'6.',6,0);                                                                               

call execute('create ' + CD[skn] +'  from ' + C[skn] + ';');                                                                               

call execute('append from ' + C[skn] + ';') ;                                                                               

call execute('close ' + CD[skn] + ';' ); 

After doing this when want to combine all these data sets I got out of memory error in log window, I am using following command to combine data sets

Data CCD;                                                                               

set CD_:;                                                                               

rename COL1=C;                                                                               

run; 

Please help

Super User
Posts: 5,255

Re: Out of Memory Error

Posting the vital part of the log would help.

Also, what are you trying to achieve?

Sounds tedious to create all those data sets, and the combine them again. Perhaps there's another less resource intense way to do this.

Data never sleeps
Frequent Contributor
Posts: 84

Re: Out of Memory Error

Actually It is a long SAS program that I am working on it, in short, I am doing 100 simulations and for each simulated data set I did 1000 bootstrap sampling (from higher order cross over designs for multivariate bioequivalence) there for each combination say simulation 1 and bootstrap sample 1 I need to create a data set I called it CD_1 therefore I needed to create data set CD_1 to CD_1001000.

Log error

497  Data CCD;

498  set CD_:;

ERROR: Out of memory.

ERROR: Out of memory.

ERROR: Out of memory.

ERROR: Out of memory.

Super User
Posts: 3,101

Re: Out of Memory Error

It appears your program is producing hundreds of datasets then trying to read them all in one DATA step. I'm not surprised you are running out of memory.

I suggest you consider a different approach that doesn't involve hundred's of datasets. I've never struck a problem that requires so many.

Unfortunately I've no idea what you are trying to do with your code as you haven't described your problem so I can't suggest a better way.

Frequent Contributor
Posts: 84

Re: Out of Memory Error

Actually It is a long SAS program that I am working on it, in short, I am doing 100 simulations and for each simulated data set I did 1000 bootstrap sampling (from higher order cross over designs for multivariate bioequivalence) there for each combination say simulation 1 and bootstrap sample 1 I need to create a data set I called it CD_1 therefore I needed to create data set CD_1 to CD_1001000.

Log error

497  Data CCD;

498  set CD_:;

ERROR: Out of memory.

ERROR: Out of memory.

ERROR: Out of memory.

ERROR: Out of memory.

Super User
Super User
Posts: 7,392

Re: Out of Memory Error

I would suggest you post, in the statistical forum area, what you are attempting to do.  The people there may be able to help you define a proper model for the task.  Alternatively, post your code here, some sample data and an explanation of what you are trying to achieve.  Creating that number of datasets is obviously not the way to proceed.

Super User
Posts: 9,671

Re: Out of Memory Error

Try proc append .

Super User
Posts: 6,928

Re: Out of Memory Error

You're trying to feed 100.000 operations into the input buffer of the main interpreter, which causes an overflow.

Rethink your basic design, and make it less complicated.

---------------------------------------------------------------------------------------------
Maxims of Maximally Efficient SAS Programmers
Super User
Posts: 5,074

Re: Out of Memory Error

Also note, are you creating unique data sets?  Can your logic distinguish s=1 and k=11 from s=11 and k=1?  You might need to use something like sk=catt(put(s,z3.), put(k,z4.));

Ask a Question
Discussion stats
  • 8 replies
  • 569 views
  • 0 likes
  • 7 in conversation