Out of Memory Issue using SAS Datastep.

Reply
New Contributor
Posts: 3

Out of Memory Issue using SAS Datastep.

We are using Base SAS 9.1.3 in Windows environment in our company.

We are performing some data step processing on really huge datasets. Even though we have around 800 GB of free space in the disk and the datasets being processed are of few GB's still we are getting the below Error while executing the Data step code:

ERROR: Out of memory.

FATAL: Error occurred in the code generation subsystem during program load.

NOTE: The SAS System stopped processing this step because of errors.


Can anyone help us regarding how we can fix this issue ?

Please let us know what is the best and the shortest way to resolve this issue.

Many thanks in Advance.

Super Contributor
Posts: 644

Re: Out of Memory Issue using SAS Datastep.

This is out of memory on your system, not disk.  What version of Windows and how much memory is installed?  How much memory is available when you start SAS?  Can you post the code so we can judge its efficiency? 

You may need to shut down other applications while SAS is running.  Memory errors are not common if you have 2 G or more (3 G max in 32 bit versions of Windows).

Often a problem like this can be resolved by rebooting.  Is your antivirus up to date?

Richard

PROC Star
Posts: 7,363

Re: Out of Memory Issue using SAS Datastep.

Super Contributor
Posts: 644

Re: Out of Memory Issue using SAS Datastep.

Art

Saurabh reports they are using 9.1.3, where this problem is supposed to have been fixed.

We need to see the code

Richard

New Contributor
Posts: 3

Re: Out of Memory Issue using SAS Datastep.

Hi Richard,

Thanks for your reply. I tried rebooting the server on which I was running the code. It didn't solve the issue. Even the antivirus is fine. Please find below the code which is failing and giving the error. The datasets being worked upon is having more then 10 million records with 650 columns. This is being used to generate certain probabilities.


data &target._rspndt_p;

      set &target._rspndt_d;

      by %if %lowcase(&target.) eq %str(pool_ds) %then %do;

  pool_tgt_demo

   %end;

   %else %if %lowcase(&target.) eq %str(competitor_ds) %then %do;

  competitive_brand_ch

  %end;

  panelmember_id &time_by_var. &client_summary_by_vars. &demographic_level._tgt_demo &demographic_level._spot_num_pnlmbr;       

      if first.&demographic_level._tgt_demo then do;

        _0 = .;

        _1 = .;

        _0 = 1 - viewing_fraction;

        _1 =     viewing_fraction;

        %if &maxspots. > 1 %then %do a = 2 %to &maxspots.;

           _%eval(&a.) = .; 

        %end;      

      end;

      else do;

        %do b = 2 %to &maxspots.;

          if &demographic_level._spot_num_pnlmbr = &b. then do;

            retain _0 _1;

            retain _%eval(&b. - 1);       

            _&b.=  _%eval(&b. - 1) * viewing_fraction; 

            

            %let d = %eval(&b. - 1);

            %if &b > 2 %then %do %while(&d. >= 2);

                retain _%eval(&d.-1);

           _&d. = (_%eval(&d.-1) * viewing_fraction) + ((_&d.*(1-viewing_fraction))); 

              %let d = %eval(&d.-1);      

      %end; 


      _1  = (_0 *      viewing_fraction) + ((_1 * (1-viewing_fraction)));

            _0  =  _0 * (1 - viewing_fraction);      

          end;

       %end;

      end; 

    run;



Let me know in case any other detail is required.

Thanks for all your help.

Thanks & Regards

Saurabh

Super User
Posts: 5,082

Re: Out of Memory Issue using SAS Datastep.

This may not help, but it surely can't hurt.

Your code generates probably hundreds, perhaps thousands of RETAIN statements most of which are redundant.  You would probably do better by removing all of the RETAIN statements, and replacing them with just one right after the BY statement is complete:

retain _0 - _%eval(&maxspots - 1);

Good luck.

Super User
Posts: 5,256

Re: Out of Memory Issue using SAS Datastep.

Have you used MPRINT to see if there any other "macro leaks" in the logic?

Another option is to run a non macro version of the code in debug mode to see if anything unpredictable is happening.

Try to analyze the resource consumption when running your program, try to turn on PerfMon, or at least Task Manager and refreshes to the file allocations in the file system(s). Also running your program with a obs= limitation, together with options fullstimer, might give you some hints about bottlenecks.

Data never sleeps
Ask a Question
Discussion stats
  • 6 replies
  • 412 views
  • 0 likes
  • 5 in conversation