Desktop productivity for business analysts and programmers

How to save disk space when executing a complex macro structure?

Reply
Occasional Contributor
Posts: 19

How to save disk space when executing a complex macro structure?

I have inherited a complex macro structure which nested loops used to work fine with 70 iterations of the outermost loop.

Now I have 86 iterations and sas colapses (I/O -erros) after 20 iterations.

 

I solve this in only executing 15 iterations at a time:

 


%loopy_a(beginloop=1, endloop= 5,libi  = VALI_OOS);
%put xxxxx '6 - > 10';
%loopy_a(beginloop=6, endloop= 10,libi = VALI_OOS);
%put xxxxx '11 - > 15';
%loopy_a(beginloop=11, endloop= 15,libi = VALI_OOS);
%put xxxxx '16 - > 20';
%loopy_a(beginloop=16, endloop= 20,libi = VALI_OOS);
%put xxxxx '21 - > 25';
%loopy_a(beginloop=21, endloop= 25,libi = VALI_OOS);
%put xxxxx '26 - > 30';
%loopy_a(beginloop=26, endloop= 30,libi = VALI_OOS);
%put xxxxx '31 - > 35';
%loopy_a(beginloop=31, endloop= 35,libi = VALI_OOS);
%put xxxxx '36 - > 40';
%loopy_a(beginloop=36, endloop= 40,libi = VALI_OOS);

This solution is all but elegant. I would very much prefer to execute loopy_a from begin_loop =1 to end_loop = 86 at a time.

I suspect the amount of data handled is not the problem, but the looped expansion of the macro structure.

Is there a way to get around this? Can I ask sas somehow not to expand the macro structure? Or am I having another problem?

 

 

Super User
Super User
Posts: 6,842

Re: How to save disk space when executing a complex macro structure?

[ Edited ]

There is not a general answer to this question. It depends on what the macro is actually doing.

It might be simple to recode the macro to work in a way that does not cause I/O errors, but we can't even make many suggestions without knowing what the macro is doing.

Occasional Contributor
Posts: 19

Re: How to save disk space when executing a complex macro structure?

You aked for it :-).

 

The macro runs a loop of sub macros that also run loops that create about 300 small temporary files in the work directory:

 


%macro loopy_a(beginloop=, endloop= ,libi=);

%do aaaa = &beginloop %to &endloop;
%let bbbb = %eval(&aaaa+13);

%Let FirstMonth = &aaaa;
%Let LastMonth = &bbbb;


%put loop &aaaa. 1;

%Select_and_Sort(libi=&libi.,acht10='no',FILTERT=(TYP ne ''),TYP=G);
%put xxxxx loop &aaaa mit beginloop = &beginloop  endloop = &endloop 
%Inner_Loop1(libi=&libi.,acht10='no',FILTERT=(TYP ne ''),TYP=G);
[inner_loop 2-18]
%Inner_Loop19(libi=&libi.,acht10='no',FILTERT=(TYP ne ''),TYP=G);
%Inner_Loop_Final(libi=&libi.,acht10='no',FILTERT=(TYP ne ''),TYP=G);

%end;


%mend;

Do you need more details on the inner loops?

 

Trusted Advisor
Posts: 1,784

Re: How to save disk space when executing a complex macro structure?

Also, within the macro you can delete some temporary datasets when they are no longer needed.

 

Are you sure it is I/O that causes SAS to crash? How do you know?

Occasional Contributor
Posts: 19

Re: How to save disk space when executing a complex macro structure?

[ Edited ]

Could it be helpful, to save the temporary files somewhere else, maybe even on a different server?

 

I am not sure about the REAL cause of the errors.

The I/O errors are jsut the first that appear.

 

Valued Guide
Posts: 3,208

Re: How to save disk space when executing a complex macro structure?

[ Edited ]

The macro as generated text code is not the issue it is  compilded than fiexed in size.
The macro is generating soms SAS code and the problem is somwhere there. 

 

It seems the clean-up on every turn is not done inside the looping of the loop there  but after that.

Fix that ....

For fixing the macro you will need to go to the owner of the macro. That is your local company. Maybe your are the owner yourself now.
Get to the macro code and analyze that one instead of just using/calling it.   

---->-- ja karman --<-----
Occasional Contributor
Posts: 19

Re: How to save disk space when executing a complex macro structure?

So can I stop sas somehow from compiling the big snowball?

Can I somehow implement a clean up within the loop?

Would this clean up be about the compiled code?

Can the data in work space be the problem? - Are there to many files in the work space?

Would it help to store the temporary files in another libary on the server, not locally?

Thanks.

 

 

 

 

 

Respected Advisor
Posts: 4,131

Re: How to save disk space when executing a complex macro structure?

Without seing your code and log it's not possible to give you real advice. If you can provide both to us (as attachments).

 

SAS WORK tables are physically written to a file location. They get automatically deleted when the SAS session closes. It's also possible to delete some or all tables in WORK via code.

 

I've seen in rare cases I/O error occur when there was no more space on the disk. Given that you get this error based on the number of "loops" it could be that every single loop creates huge tables in WORK and that you simply run out of disk space.

 

One thing you could try: On top of your code before you call the macros execute "options compress=yes;"

 

That's not the real solution but may-be it could get you over this hurdle for the moment.

 

If it's a space problem with WORK then the "real solution" would be:

-  get more disk space for WORK

-  or even better: amend your code so that it doesn't use that much space

 

 

Ask a Question
Discussion stats
  • 7 replies
  • 389 views
  • 3 likes
  • 5 in conversation