topic Re: Is there a tool to transform a lot of macro files to the Stored Compiled Macro program? in SAS Programming

Is there a tool to transform a lot of macro files to the Stored Compiled Macro program?

jjjch — Thu, 27 Feb 2020 04:04:58 GMT

I save each macro function into a SAS file and put them all in my_macro_folder.

/*my_macro1.sas file*/
%macro my_macro1();
...
%mend;

/*my_macro2.sas file*/
%macro my_macro2();
...
%mend;

And below is the way that I call those macros.

options sasautos=("my_macro_folder" sasautos);
%my_macro1()
%my_macro2()
...

As my program evolves and adds more functions, the number of macro functions increases, and my program becomes slower and slower. Recently I noticed that it took 13 minutes to run the first time, then 8 minutes to run the second time for the exact same input.

What are the possible ways to speed up the program?

Looks like switching to the stored compiled macro programs is a way of improving the efficiency. Then I will need to add " / store source" to each macro function and combine them into one file like below. Is there a tool to do it automatically? Because I would like to keep using the sasautos way to develop and test my program while only using the stored compiled macro programs in the production environment, a tool to automatically transform all my macros into these stored compiled macro programs would save developers like me a lot of time.

libname mylib 'my_macro_folder';
options mstored sasmstore=mylib;

/*my_macro1.sas file*/
%macro my_macro1() / store source;
 ...
%mend;

/*my_macro2.sas file*/
%macro my_macro2() / store source;
 ...
%mend;

Thank you!

Re: Is there a tool to transform a lot of macro files to the Stored Compiled Macro program?

Patrick — Thu, 27 Feb 2020 04:31:06 GMT

You can store macros in any catalog. It doesn't need to be the mstore.

For your use case I'd actually would go for a project specific catalog using the technique described here under section "Using SAS Catalogs as Autocall Libraries"

You then could simply write a program which scans the folder with the .sas macro code and executes them one by one (just compiling into your project specific catalog).

Scanning the file system to find not yet compiled macros can add a bit of time. The 5 minutes you observe feel very excessive though.

If there is really no other reason for the difference between first and 2nd run (like cached data, or less data processing due to data already loaded in the first run) then I'd also investigate if there is an I/O or latency issue with the file system where you store your macro code.

Re: Is there a tool to transform a lot of macro files to the Stored Compiled Macro program?

Tom — Thu, 27 Feb 2020 04:41:44 GMT

Hard to tell what your issue is, but it sounds like you are saying that as your program gets larger it take longer to run. Seems normal to me.

Storing the compiled macro will not really increase performance (unless you file system is terrible). And it will cause all kinds of maintenance nightmares. But if you did want to do it just run the code that defines the macros. So run once (or when ever the source code changes:

libname mylib 'my_macro_folder';
options mstored sasmstore=mylib;
filename mysource 'my_macro_folder_with_source_code';
%include mysource(*);

Then in your real programs you just need the first two lines.

Re: Is there a tool to transform a lot of macro files to the Stored Compiled Macro program?

jjjch — Thu, 27 Feb 2020 16:18:13 GMT

Hi Patrick, thank you for your reply. Looks like I still need to add " / store " to my macros and compile the macros one by one.

And I hope there is a real example of "Using SAS Catalogs as Autocall Libraries" in the document to show us how it works.

Re: Is there a tool to transform a lot of macro files to the Stored Compiled Macro program?

jjjch — Thu, 27 Feb 2020 16:34:28 GMT

Hi Tom, thank you for your reply. I got the below error for the statement.

%include mysource(*);

ERROR: Cannot %INCLUDE member (*) in the aggregate MYSOURCE.

%include mysource(my_macro1, my_macro2);

Works though.

Re: Is there a tool to transform a lot of macro files to the Stored Compiled Macro program?

Tom — Thu, 27 Feb 2020 16:33:58 GMT

If it doesn’t work without modifying the code you could also just try running the macro definitions using “normal” settings. That will store the compiled macros in a catalog in WORK library. Normally WORK.SASMACR, but if using EG or SAS/Studio to launch SAS it might be named WORK.SASMAC1 instead. You can then use PROC CATALOG to copy the compiled macros to your permanent catalog.

Let me state again that normally the compiling of the macro via SASAUTO is NOT a performance issue so trying to convert to using pre-compiled macros will normally not result in any change in performance.

Re: Is there a tool to transform a lot of macro files to the Stored Compiled Macro program?

jjjch — Thu, 27 Feb 2020 17:08:59 GMT

Hi Tom,

I have about 250 macro functions. I want to know how many seconds they need to be compiled. But looks like it is not easy to find out.

Because I may need to schedule my program to run every 15 minutes or every 10 minutes, and I hope using pre-compiled macros could save me some time.

Thanks.

Re: Is there a tool to transform a lot of macro files to the Stored Compiled Macro program?

RichardDeVen — Sat, 07 Mar 2020 17:04:12 GMT

By using auto-call system you are simplifying the utilization of the macro, but masking the compilation time with the invocation.

There are a couple of options to see what auto-call and macro is doing

MAUTOCOMPLOC
MCOMPILENOTE
MEXECNOTE
MEXECSIZE

The compilation time of macros should be static.

The invocation timing is highly variant and dependent on items such as:

Control data affecting code generation
Quality of generated code
Data upon which the code operates
Outputs and reports that the code creates
Operating system factors impacting reading and writing data and ODS
- Network share ?
- File contention ?
- Thread reliance
- Inter-OS file sharing ?

Re: Is there a tool to transform a lot of macro files to the Stored Compiled Macro program?

SASKiwi — Sat, 07 Mar 2020 21:05:05 GMT

IMO I think you are looking in the wrong place to improve program performance. I use AUTOCALL macros a lot and I've never seen a significant performance hit as it grows. I suggest looking at the IO performance. The fact that your program runs a lot slower first time is indicative of slow IO and data caching could be the reason it speeds up.

You can easily test the overhead of compiling your macros by %INCLUDEing them first, then timing your program runs after that. If that doesn't change your overall run times then your performance problems are elsewhere.