03-02-2013 02:42 AM
We are using Base SAS 9.1.3 in Windows environment in our company.
We are performing some data step processing on really huge datasets. Even though we have around 800 GB of free space in the disk and the datasets being processed are of few GB's still we are getting the below Error while executing the Data step code:
FATAL: Error occurred in the code generation subsystem during program load.
NOTE: The SAS System stopped processing this step because of errors.
Can anyone help us regarding how we can fix this issue ?
Please let us know what is the best and the shortest way to resolve this issue.
Many thanks in Advance.
03-02-2013 05:07 AM
This is out of memory on your system, not disk. What version of Windows and how much memory is installed? How much memory is available when you start SAS? Can you post the code so we can judge its efficiency?
You may need to shut down other applications while SAS is running. Memory errors are not common if you have 2 G or more (3 G max in 32 bit versions of Windows).
Often a problem like this can be resolved by rebooting. Is your antivirus up to date?
03-05-2013 10:53 AM
Thanks for your reply. I tried rebooting the server on which I was running the code. It didn't solve the issue. Even the antivirus is fine. Please find below the code which is failing and giving the error. The datasets being worked upon is having more then 10 million records with 650 columns. This is being used to generate certain probabilities.
by %if %lowcase(&target.) eq %str(pool_ds) %then %do;
%else %if %lowcase(&target.) eq %str(competitor_ds) %then %do;
panelmember_id &time_by_var. &client_summary_by_vars. &demographic_level._tgt_demo &demographic_level._spot_num_pnlmbr;
if first.&demographic_level._tgt_demo then do;
_0 = .;
_1 = .;
_0 = 1 - viewing_fraction;
_1 = viewing_fraction;
%if &maxspots. > 1 %then %do a = 2 %to &maxspots.;
_%eval(&a.) = .;
%do b = 2 %to &maxspots.;
if &demographic_level._spot_num_pnlmbr = &b. then do;
retain _0 _1;
retain _%eval(&b. - 1);
_&b.= _%eval(&b. - 1) * viewing_fraction;
%let d = %eval(&b. - 1);
%if &b > 2 %then %do %while(&d. >= 2);
_&d. = (_%eval(&d.-1) * viewing_fraction) + ((_&d.*(1-viewing_fraction)));
%let d = %eval(&d.-1);
_1 = (_0 * viewing_fraction) + ((_1 * (1-viewing_fraction)));
_0 = _0 * (1 - viewing_fraction);
Let me know in case any other detail is required.
Thanks for all your help.
Thanks & Regards
03-05-2013 11:26 AM
This may not help, but it surely can't hurt.
Your code generates probably hundreds, perhaps thousands of RETAIN statements most of which are redundant. You would probably do better by removing all of the RETAIN statements, and replacing them with just one right after the BY statement is complete:
retain _0 - _%eval(&maxspots - 1);
03-05-2013 03:18 PM
Have you used MPRINT to see if there any other "macro leaks" in the logic?
Another option is to run a non macro version of the code in debug mode to see if anything unpredictable is happening.
Try to analyze the resource consumption when running your program, try to turn on PerfMon, or at least Task Manager and refreshes to the file allocations in the file system(s). Also running your program with a obs= limitation, together with options fullstimer, might give you some hints about bottlenecks.