BookmarkSubscribeRSS Feed
NKormanik
Barite | Level 11

The SAS program I'm running involves Proc G3Grid, then a Data step.  These two repeat over and over, say, 100 times.

This process gets through 3 iterations of the above just fine, no problems, then it can't do any more after 3.  Here is the error message:

ERROR: Memory allocation failure.

ERROR: The SAS System stopped processing this step because of insufficient memory.

Since 3 got done successfully, something seems to then get retained by the memory, and accumulates, until all memory is filled.

Is it possible to put a command between the iterations to clear memory?

Any other ideas you might have would be greatly appreciated.

Thanks.

Nicholas Kormanik

8 REPLIES 8
LinusH
Tourmaline | Level 20

No experience with g3grid.

Adding options fullstimer; can help you understand what consumes memory.

Is one iteration 1000 repeats of Proc g3grid and a data step?

If so, how many iteration do wish to do?

Maybe you could have a new session for each 3 iterations?

Or maybe you could change your program logic to reduce the number of repeats/iterations...?

Data never sleeps
NKormanik
Barite | Level 11

Here's the code for the first iteration:

proc g3grid data=nicholas.data out=nicholas._20101_20102_ ;

  grid '20101'n * '20102'n = '50501'n '50502'n '50503'n '50504'n '50505'n '50508'n '50401'n '50402'n '50403'n '50404'n '50405'n '50408'n / spline smooth = .05 naxis1 = 21 naxis2 = 21  ;

  run ;

  quit ;

  data nicholas._20101_20102_ ;

  set nicholas._20101_20102_ ;

  Combo = '20101_20102' ;

  label '20101'n = Var_1 ;

  label '20102'n = Var_2 ;

  rename '20101'n = Var_1 ;

  rename '20102'n = Var_2 ;

  run ;

  quit ;

  data nicholas._20101_20102_ ;

  retain Combo ;

  set nicholas._20101_20102_ ;

  run ;

  quit ;

=====

There are 16,000 of such iterations in total.  Having a new session for every, say, 3 iterations would be a lot of work.  I'm hoping there will be some easier way.  One possibility I'm going to try is to make the data set smaller.

data_null__
Jade | Level 19

Is this in a macro loop?  Anyway you're going to have to do something other than 16000 calls to a proc/data/data.  Show more explain more and start from the beginning.

LinusH
Tourmaline | Level 20

For starters,  you can skip the second data step,  use retain in the first step.Have you looked at the possibility to use BY group processing in g3grid? If applicable it could reduce the number of iterations. Again, what does fullstimer show?  what is your memsize setting?  How much physical memory do you have?

Data never sleeps
NKormanik
Barite | Level 11

Not sure about fullstimer.  But, I cut my total data file to one-tenth (10%) of its size -- from 100,000 observations, to 10,000 observations -- and SAS was able to run the above program.  However each iteration takes 20 minutes!!  With 16,000 iterations needed.

Memsize in SAS Options Environment > System administration > Memory > Memsize = 2147483648

Memory in computer is 4Gb.  Windows 7, 64-bit.  SAS 9.3, 32-bit.

So..., the best suggestion seems to be to rewrite the program??

LinusH
Tourmaline | Level 20

I think it's safe to say you need to rewrite your program and/or reorganize your data.

Since I don't understand what/why you are doing, can't help you with how.

options fullstimer;

Data never sleeps
NKormanik
Barite | Level 11

After experimentation, seems memory usage and time needed per iteration is HUGELY contingent upon the size of the data set.

If the data set is small -- around 5000 observations or less -- SAS will chug along and get the job done.  A few minutes per iteration.

No choice, then, but to sub-divide data set.

Curious to have access to SAS on a university mainframe, and see what happens.  Would the entire thing be done in a flash??

Reeza
Super User

100,000 records is nothing. SAS should be able to deal with it easily, so it implies something very inefficient about the program or the G3GRID procedure.

I'd contact tech support or provide more information to the boards.

sas-innovate-2024.png

Join us for SAS Innovate April 16-19 at the Aria in Las Vegas. Bring the team and save big with our group pricing for a limited time only.

Pre-conference courses and tutorials are filling up fast and are always a sellout. Register today to reserve your seat.

 

Register now!

What is Bayesian Analysis?

Learn the difference between classical and Bayesian statistical approaches and see a few PROC examples to perform Bayesian analysis in this video.

Find more tutorials on the SAS Users YouTube channel.

Click image to register for webinarClick image to register for webinar

Classroom Training Available!

Select SAS Training centers are offering in-person courses. View upcoming courses for:

View all other training opportunities.

Discussion stats
  • 8 replies
  • 4501 views
  • 0 likes
  • 4 in conversation