topic Re: SAS Always freezing when running large scale program? in SAS Programming

SAS Always freezing when running large scale program?

AllSoEasy — Wed, 24 Apr 2013 18:23:00 GMT

I have a fairly large program that is about 11,000 lines of macro definition and then a ~50 line triple nested loop which essentially runs the whole program by calling to all of the macros, the inner-most loop iterates from 1 to 7, the outter most loop iterates from 1 to 3, and middle loop iterates anywhere from 1 to 33 times. I.e:

do i = 1 to 3;

do j = 1 to 33;

do k = 1 to 7;

if it is run with the second loop as do j = 1 to 2; or do j = 1 to 3; it runs okay, but when it gets much more than this SAS freezes every time, it will just say "Running: Exec_All_Models - Running PROC SGPLOT" or something at the bottom frozen, and it will not continue executing, I have to kill the process via task manager and restart SAS every time. It is producing lots of output files including pdfs etc, so I expect it to take a bit of time to run, but it shouldn't just completely freeze after it's gotten past a certain point, it's re-running all the same code just with different data sets. I'm not sure what to do, is there any way to get SAS to handle this or do I just have to always manually edit the loop dimensions and run the program in pieces every time?

Thanks!

Re: SAS Always freezing when running large scale program?

AllSoEasy — Wed, 24 Apr 2013 18:43:55 GMT

I've attached an example log of it freezing, if that may help...There doesn't seem to be fatal errors present in the log

Re: SAS Always freezing when running large scale program?

ballardw — Wed, 24 Apr 2013 20:28:55 GMT

You may be genreating more output than SAS output can hold in memory if generating List, HTML and PDF. I've had programs apparently hang when it was just a LOT of HTML tables that were attempting to render.

You may want to search the forum for clearing results and or listing periodically within the program where you final output for sections has been completed, maybe at the end of each "J" loop. I am assuming here that the PDFs are the actual desired output.

Re: SAS Always freezing when running large scale program?

Patrick — Thu, 25 Apr 2013 00:21:29 GMT

First thing you should do is scan your log for warnings and amend your code so that no warnings are thrown, eg: "WARNING: You must enable ODS graphics with the ODS GRAPHICS ON statement before requesting plots."

You also might want to close all ODS destinations you don't need.

You're running your code out of EG. The freezing could occur because you're generating a lot of output which EG might try to open. If not done so already "un-tick" in EG under "Tools/Options/Result General" box "Automatically open data or results when generated" and under "Tools/Options/Data General" box "Automatically open data when added to project".

Re: SAS Always freezing when running large scale program?

AllSoEasy — Thu, 25 Apr 2013 13:15:26 GMT

Yes I was aware of that warning, Patrick, I intentionally set ods graphics off, that tends to speed it up because it doesn't export all of the proc model plots to the pdf (we do in general want these plots, but I was trying to speed it up and get it to at least run without freezing).

- I did try setting those two options in Tools to disallow opening data results.

-ballardw, Yes I do want the pdf output, but all output I need is exported as pdf/xls files. I only need the files generated by ods pdf, and proc export. I do not need any of the output data or pdf results within sas that it adds to the project. I've been looking for a way to disable this.

-Thanks for the help.

Re: SAS Always freezing when running large scale program?

Patrick — Fri, 26 Apr 2013 10:11:33 GMT

So does it work now or do you still have "freezing" problems?

If my theory is correct that the freezing occurs because of the volume of results streamed back to EG then you could also try and run your job in batch using the "systask" command.

I had once a "EG freezing" problem at a site not because there was any issue with EG but the network interrupted the connection if it "thought" that a client was idle for more than 5 minutes. So when executing long running jobs I basically needed to do "something" with EG like opening a file simply to keep the connection alive.

Re: SAS Always freezing when running large scale program?

AllSoEasy — Wed, 01 May 2013 17:06:46 GMT

No, it still has those freezing problems, setting those data results options didn't yield any really noticeable difference.

As for systask I will research how to use and look into that. As for doing something with "EG," what exactly is EG?

Thanks a lot!!!

Re: SAS Always freezing when running large scale program?

Reeza — Wed, 01 May 2013 17:29:59 GMT

EG is enterprise Guide and it looks like a lot of your code is generated from EG or you're running through EG.

You have a lot of processing going on, but the datasets appear really small. I would really investigate the program to see if there were better ways to make it efficient. I'm not going to read through all of the code, but you could start on the inner loops and see if you couldnt' use some by processing instead of loops. As well, you seem to create a lot of macro variables, and I wonder if there wasn't a way around some of that.

Good luck.

Re: SAS Always freezing when running large scale program?

AllSoEasy — Wed, 01 May 2013 18:16:46 GMT

Ahh I see, yes I user SAS Enterprise Guide, so Patrick, you mean just opening a file in the SAS interface will keep it from "losing connection"/freezing?

And Reeza, yes you definitely don't want to read through the 10,000 lines of code I'm sure ;-P. However, on a serious note, I'm not sure what you mean by "use some by processing instead of loops"? I need to loop over all of the array items; I am unaware of any computational way to process a list of items without iterating over the items?

And yes I do create a lot of macro variables, but unfortunately each one of them is necessary.

My background is in computer science and was a software engineer for some years in the field, and so I am familiar with programmatic ways to increase efficiency of a program and remove unnecessary statements, but I don't think I can optimize this program much more without removing functionality.

Thanks for all of the information/suggestions!

Re: SAS Always freezing when running large scale program?

Reeza — Wed, 01 May 2013 19:01:52 GMT

I scanned your code again, and see you've used BY for merging.

Sometimes its better to transpose your data (or in your case not transpose) and process in by rather than with an array that goes across the data.

using the by will allow you to summarize things in some more automated fashions.

I'm not sure if you know about BY processing, but usually it speeds things up.

And its one of the reasons that SAS is so powerful.

Here's a good paper on BY processing.

http://www.lexjansen.com/wuss/2008/ess/ess11.pdf

Re: SAS Always freezing when running large scale program?

SASKiwi — Wed, 01 May 2013 20:14:55 GMT

With a problem like yours I would be tempted to try running your program as a batch job. If your batch job still has the "freeze" problem than you can be sure your issues are to do with the code and how it is running on the SAS server.

If it doesn't freeze then the problem is specific to the EG environment and how your code interacts with it.

Re: SAS Always freezing when running large scale program?

MumSquared — Mon, 04 Nov 2013 23:25:34 GMT

I am not sure this helps but a few years ago I was producing thousands of rft output files and found there was a windows handle limit on the output files of about 10,000.

I had to break the program into pieces and run each piece in separate SAS sessions.

Good luck!

Re: SAS Always freezing when running large scale program?

scmebu — Tue, 05 Nov 2013 15:49:21 GMT

It might help to run SAS with the -unbuflog option. Using it, you're more likely to see the last executed step or procedure. Production of the log might take longer, especially given a large job with copious output, but the log is less likely to be truncated.

Re: SAS Always freezing when running large scale program?

Tom — Tue, 05 Nov 2013 17:11:25 GMT

So it looks like you are pushing a LOT of code onto the stack using CALL EXECUTE in the data step at about line 10,080 in your log. There are timing issues that can occur when you try to call a macro inside of a CALL EXECUTE(). One solution is to make sure to use %NRSTR() in the text that is passed to CALL EXECUTE so that the macro call is pushed onto the stack instead of the code that the macro generates. Example:

10092	Set_Product_Specifics_call = '%nrstr(%Set_Product_Specifics)(dep_rate='\|\|deposit_rates\|\|')';
10093	call execute(Set_Product_Specifics_call);

Personally I find that it is much easier to use the PUT statement to generate the code to a file and then %INC the generated file instead of using CALL EXECUTE. Usually eliminates a lot of string concatenation and number to text conversions.

Re: SAS Always freezing when running large scale program?

Tom — Tue, 05 Nov 2013 17:38:59 GMT

Something like this might work for that data step.

/*This data step runs the entire model execution & data exportation process, that is

data for accounts, market rates, and shock scenarios is established here, and then

all subsequent macro calls stem from this data step:

file code temp;

data _null_;

file code ;

set merge_shorts_deposits;

/*Create array for the 3 things we are iterating over - deposit accounts, market

rates, and [short rate] shock scenarios */

array shocks (7) shock_1-shock_7;

array short_rates_main (3) Short_Rate_1-Short_Rate_3;

array deposit_rates(33) Deposit_Rate_1-Deposit_Rate_33;

do i = 1 to 1;

do j = 6,20,21,27,30,32,33 ;

put '%Set_Product_Specifics(dep_rate=' deposit_rates ')';

do k = 1 to 7;

put '%Create_Fundamental_Datasets'

/ '(short_rate=' short_rates_main

/ ',dep_rate=' deposit_rates

/ ',shock_scenario=' shocks

/ ')'

;

put '%Execute_Old_Model(shock_scenario=' shocks ')';

put '%Execute_All_Models(shock_scenario=' shocks ')';

end;

put '%Export_All_Output'

/ '(pdf_file=&rootnode\' deposit_rates +(-1) '_vs' short_rates_main

/ ',forecast_file=&rootnode\Forecasts_' deposit_rates +(-1) "_vs" short_rates_main

/ ',stats_file=&rootnode\ModelStatistics_' deposit_rates +(-1) "_vs" short_rates_main

/ ',short_rate=' short_rates_main

/ ',dep_rate=' deposit_rate

/ ')'

;

put '%Calculate_Ex_Posts;';

put '%Export_Old_Output'

/ '(pdf_file=&rootnode\' deposit_rates +(-1) '_vs' short_rates_main

/ ',forecast_file=&rootnode\Forecasts_' deposit_rates +(-1) "_vs" short_rates_main

/ ',stats_file=&rootnode\ModelStatistics_' deposit_rates +(-1) "_vs" short_rates_main

/ ',short_rate=' short_rates_main

/ ',dep_rate=' deposit_rate

/ ')'

;

put '%Export_All_Estimates'

/ '(dep_rate=' deposit_rates

/ ',short_rate=' short_rates_main

/ ')'

;

end;

run;

%let rootnode=C:\Documents and Settings\ETRORMC\My Documents\SAS_Test_Output ;

%include code / source2 ;