topic Re: Out of Memory Error in SAS Data Management

Out of Memory Error

Rasheed — Wed, 26 Aug 2015 08:47:17 GMT

I am working on SAS program where I needed to create many data sets and I have successfully created data sets from CD_1 to CD_1001000 by using following sas command

CD='CD_1':'CD_1001000';

do s=1 to 100;

do k = 1 to 1000;

sk=catt(s,k);

skn=inputn(sk,'6.',6,0);

call execute('create ' + CD[skn] +' from ' + C[skn] + ';');

call execute('append from ' + C[skn] + ';') ;

call execute('close ' + CD[skn] + ';' );

After doing this when want to combine all these data sets I got out of memory error in log window, I am using following command to combine data sets

Data CCD;

set CD_:;

rename COL1=C;

run;

Please help

Re: Out of Memory Error

LinusH — Wed, 26 Aug 2015 08:53:11 GMT

Posting the vital part of the log would help.

Also, what are you trying to achieve?

Sounds tedious to create all those data sets, and the combine them again. Perhaps there's another less resource intense way to do this.

Re: Out of Memory Error

Rasheed — Wed, 26 Aug 2015 09:03:26 GMT

Actually It is a long SAS program that I am working on it, in short, I am doing 100 simulations and for each simulated data set I did 1000 bootstrap sampling (from higher order cross over designs for multivariate bioequivalence) there for each combination say simulation 1 and bootstrap sample 1 I need to create a data set I called it CD_1 therefore I needed to create data set CD_1 to CD_1001000.

Log error

497 Data CCD;

498 set CD_:;

ERROR: Out of memory.

Re: Out of Memory Error

SASKiwi — Wed, 26 Aug 2015 09:03:57 GMT

It appears your program is producing hundreds of datasets then trying to read them all in one DATA step. I'm not surprised you are running out of memory.

I suggest you consider a different approach that doesn't involve hundred's of datasets. I've never struck a problem that requires so many.

Unfortunately I've no idea what you are trying to do with your code as you haven't described your problem so I can't suggest a better way.

Re: Out of Memory Error

Rasheed — Wed, 26 Aug 2015 09:10:48 GMT

Log error

497 Data CCD;

498 set CD_:;

ERROR: Out of memory.

Re: Out of Memory Error

RW9 — Wed, 26 Aug 2015 09:36:59 GMT

I would suggest you post, in the statistical forum area, what you are attempting to do. The people there may be able to help you define a proper model for the task. Alternatively, post your code here, some sample data and an explanation of what you are trying to achieve. Creating that number of datasets is obviously not the way to proceed.

Re: Out of Memory Error

Kurt_Bremser — Wed, 26 Aug 2015 11:48:14 GMT

You're trying to feed 100.000 operations into the input buffer of the main interpreter, which causes an overflow.

Rethink your basic design, and make it less complicated.

Re: Out of Memory Error

Ksharp — Wed, 26 Aug 2015 12:09:58 GMT

Try proc append .

Re: Out of Memory Error

Astounding — Wed, 26 Aug 2015 12:41:17 GMT

Also note, are you creating unique data sets? Can your logic distinguish s=1 and k=11 from s=11 and k=1? You might need to use something like sk=catt(put(s,z3.), put(k,z4.));