10-20-2015 12:20 PM
I have inherited a complex macro structure which nested loops used to work fine with 70 iterations of the outermost loop.
Now I have 86 iterations and sas colapses (I/O -erros) after 20 iterations.
I solve this in only executing 15 iterations at a time:
%loopy_a(beginloop=1, endloop= 5,libi = VALI_OOS); %put xxxxx '6 - > 10'; %loopy_a(beginloop=6, endloop= 10,libi = VALI_OOS); %put xxxxx '11 - > 15'; %loopy_a(beginloop=11, endloop= 15,libi = VALI_OOS); %put xxxxx '16 - > 20'; %loopy_a(beginloop=16, endloop= 20,libi = VALI_OOS); %put xxxxx '21 - > 25'; %loopy_a(beginloop=21, endloop= 25,libi = VALI_OOS); %put xxxxx '26 - > 30'; %loopy_a(beginloop=26, endloop= 30,libi = VALI_OOS); %put xxxxx '31 - > 35'; %loopy_a(beginloop=31, endloop= 35,libi = VALI_OOS); %put xxxxx '36 - > 40'; %loopy_a(beginloop=36, endloop= 40,libi = VALI_OOS);
This solution is all but elegant. I would very much prefer to execute loopy_a from begin_loop =1 to end_loop = 86 at a time.
I suspect the amount of data handled is not the problem, but the looped expansion of the macro structure.
Is there a way to get around this? Can I ask sas somehow not to expand the macro structure? Or am I having another problem?
10-20-2015 12:47 PM - edited 10-20-2015 12:48 PM
There is not a general answer to this question. It depends on what the macro is actually doing.
It might be simple to recode the macro to work in a way that does not cause I/O errors, but we can't even make many suggestions without knowing what the macro is doing.
10-21-2015 03:33 AM
You aked for it :-).
The macro runs a loop of sub macros that also run loops that create about 300 small temporary files in the work directory:
%macro loopy_a(beginloop=, endloop= ,libi=); %do aaaa = &beginloop %to &endloop; %let bbbb = %eval(&aaaa+13); %Let FirstMonth = &aaaa; %Let LastMonth = &bbbb; %put loop &aaaa. 1; %Select_and_Sort(libi=&libi.,acht10='no',FILTERT=(TYP ne ''),TYP=G); %put xxxxx loop &aaaa mit beginloop = &beginloop endloop = &endloop %Inner_Loop1(libi=&libi.,acht10='no',FILTERT=(TYP ne ''),TYP=G); [inner_loop 2-18] %Inner_Loop19(libi=&libi.,acht10='no',FILTERT=(TYP ne ''),TYP=G); %Inner_Loop_Final(libi=&libi.,acht10='no',FILTERT=(TYP ne ''),TYP=G); %end; %mend;
Do you need more details on the inner loops?
10-20-2015 02:16 PM
Also, within the macro you can delete some temporary datasets when they are no longer needed.
Are you sure it is I/O that causes SAS to crash? How do you know?
10-21-2015 03:26 AM - edited 10-21-2015 03:59 AM
Could it be helpful, to save the temporary files somewhere else, maybe even on a different server?
I am not sure about the REAL cause of the errors.
The I/O errors are jsut the first that appear.
10-21-2015 02:50 PM - edited 10-22-2015 02:07 PM
The macro as generated text code is not the issue it is compilded than fiexed in size.
The macro is generating soms SAS code and the problem is somwhere there.
It seems the clean-up on every turn is not done inside the looping of the loop there but after that.
Fix that ....
For fixing the macro you will need to go to the owner of the macro. That is your local company. Maybe your are the owner yourself now.
Get to the macro code and analyze that one instead of just using/calling it.
10-22-2015 04:49 AM
So can I stop sas somehow from compiling the big snowball?
Can I somehow implement a clean up within the loop?
Would this clean up be about the compiled code?
Can the data in work space be the problem? - Are there to many files in the work space?
Would it help to store the temporary files in another libary on the server, not locally?
10-22-2015 06:12 AM
Without seing your code and log it's not possible to give you real advice. If you can provide both to us (as attachments).
SAS WORK tables are physically written to a file location. They get automatically deleted when the SAS session closes. It's also possible to delete some or all tables in WORK via code.
I've seen in rare cases I/O error occur when there was no more space on the disk. Given that you get this error based on the number of "loops" it could be that every single loop creates huge tables in WORK and that you simply run out of disk space.
One thing you could try: On top of your code before you call the macros execute "options compress=yes;"
That's not the real solution but may-be it could get you over this hurdle for the moment.
If it's a space problem with WORK then the "real solution" would be:
- get more disk space for WORK
- or even better: amend your code so that it doesn't use that much space