Help using Base SAS procedures

After a number of steps SAS runs out of memory

Reply
Regular Contributor
Posts: 223

After a number of steps SAS runs out of memory

The SAS program I'm running involves Proc G3Grid, then a Data step.  These two repeat over and over, say, 100 times.

This process gets through 3 iterations of the above just fine, no problems, then it can't do any more after 3.  Here is the error message:

ERROR: Memory allocation failure.

ERROR: The SAS System stopped processing this step because of insufficient memory.

Since 3 got done successfully, something seems to then get retained by the memory, and accumulates, until all memory is filled.

Is it possible to put a command between the iterations to clear memory?

Any other ideas you might have would be greatly appreciated.

Thanks.

Nicholas Kormanik

Super User
Posts: 5,323

Re: After a number of steps SAS runs out of memory

No experience with g3grid.

Adding options fullstimer; can help you understand what consumes memory.

Is one iteration 1000 repeats of Proc g3grid and a data step?

If so, how many iteration do wish to do?

Maybe you could have a new session for each 3 iterations?

Or maybe you could change your program logic to reduce the number of repeats/iterations...?

Data never sleeps
Regular Contributor
Posts: 223

Re: After a number of steps SAS runs out of memory

Here's the code for the first iteration:

proc g3grid data=nicholas.data out=nicholas._20101_20102_ ;

  grid '20101'n * '20102'n = '50501'n '50502'n '50503'n '50504'n '50505'n '50508'n '50401'n '50402'n '50403'n '50404'n '50405'n '50408'n / spline smooth = .05 naxis1 = 21 naxis2 = 21  ;

  run ;

  quit ;

  data nicholas._20101_20102_ ;

  set nicholas._20101_20102_ ;

  Combo = '20101_20102' ;

  label '20101'n = Var_1 ;

  label '20102'n = Var_2 ;

  rename '20101'n = Var_1 ;

  rename '20102'n = Var_2 ;

  run ;

  quit ;

  data nicholas._20101_20102_ ;

  retain Combo ;

  set nicholas._20101_20102_ ;

  run ;

  quit ;

=====

There are 16,000 of such iterations in total.  Having a new session for every, say, 3 iterations would be a lot of work.  I'm hoping there will be some easier way.  One possibility I'm going to try is to make the data set smaller.

Respected Advisor
Posts: 3,780

Re: After a number of steps SAS runs out of memory

Is this in a macro loop?  Anyway you're going to have to do something other than 16000 calls to a proc/data/data.  Show more explain more and start from the beginning.

Super User
Posts: 5,323

Re: After a number of steps SAS runs out of memory

For starters,  you can skip the second data step,  use retain in the first step.Have you looked at the possibility to use BY group processing in g3grid? If applicable it could reduce the number of iterations. Again, what does fullstimer show?  what is your memsize setting?  How much physical memory do you have?

Data never sleeps
Regular Contributor
Posts: 223

Re: After a number of steps SAS runs out of memory

Not sure about fullstimer.  But, I cut my total data file to one-tenth (10%) of its size -- from 100,000 observations, to 10,000 observations -- and SAS was able to run the above program.  However each iteration takes 20 minutes!!  With 16,000 iterations needed.

Memsize in SAS Options Environment > System administration > Memory > Memsize = 2147483648

Memory in computer is 4Gb.  Windows 7, 64-bit.  SAS 9.3, 32-bit.

So..., the best suggestion seems to be to rewrite the program??

Super User
Posts: 5,323

Re: After a number of steps SAS runs out of memory

I think it's safe to say you need to rewrite your program and/or reorganize your data.

Since I don't understand what/why you are doing, can't help you with how.

options fullstimer;

Data never sleeps
Regular Contributor
Posts: 223

Re: After a number of steps SAS runs out of memory

After experimentation, seems memory usage and time needed per iteration is HUGELY contingent upon the size of the data set.

If the data set is small -- around 5000 observations or less -- SAS will chug along and get the job done.  A few minutes per iteration.

No choice, then, but to sub-divide data set.

Curious to have access to SAS on a university mainframe, and see what happens.  Would the entire thing be done in a flash??

Super User
Posts: 18,601

Re: After a number of steps SAS runs out of memory

100,000 records is nothing. SAS should be able to deal with it easily, so it implies something very inefficient about the program or the G3GRID procedure.

I'd contact tech support or provide more information to the boards.

Ask a Question
Discussion stats
  • 8 replies
  • 3241 views
  • 0 likes
  • 4 in conversation