Macro CREATION Speed

[deleted_user]

Hello. I have a question regarding the efficiency of SAS macro variables. I have more complex code however the following code makes the point:

data _null_;
%do i=1 %to 18828;
call symput(cats('testvar',&i.),&i.);
%end;
run;

filename devnull "/dev/null";

data _null_;
file devnull;
put
%do i=1 %to 18828;
"&&testvar&i."
%end;
;
stop;
run;

They above code takes ~18 seconds to run. The program is called by:

sas foo -memsize 4G -log /dev/null -MSYMTABMAX 2G -MEXECSIZE 1G

It seems terribly inefficient that this is taking 18 secs to run. I can't help to think that there is a way to make this more efficient. Any tips on how to speed macro resolution up would be greatly appreciated! Thanks! Message was edited by: PROC SUM

[darrylovia]

Everytime you create a macro variable an entry gets written in the macro symbol table either global or local depending on the scope. With your exampe you are creating 18828 global macro variables. To see what I'm writing about, go to sashelp.vmacro and you will see all 18828 macro variables.

As mentioned in this forum before, the macro facilty is a way for SAS to generate SAS code and then that generated code gets executed. In the back ground somewhere SAS is generating "a line of code" for each iteration of your loop and then executes it.

For an alternative method of looping, I would investigate CALL EXECUTE routine inside of the data step.

-Darryl

[deleted_user]

I understand the logic of how SAS handles macros. My question was is there a way to increase efficiency of them? Surely creating 18828 global macro's should not take 18 seconds. Or should it?

[deleted_user]

Also, it seems the time is eaten up not on the resolution but by the creation of the macros.

[deleted_user]

I question your need to have to create 18828 macro variables in the first place. Doing this tells me that you probably don't understand the point behind macros, don't understand SAS programming, and are trying to use macros to do the work that data steps and data sets should be doing.

In other words, are you using macros as the primary language, and then using conventional programming techniques to solve your problem, instead of letting the simplicity of SAS do it for you?

[deleted_user]

Thanks Chuck for your reply, very helpful and informative!

[deleted_user]

Ok, how about this.

1) don't rely on creating and using so many macro variables
2) Don't mix the use of regular SAS code with macro code

As an example of #2 (sort of, I can't share the code).

I had a macro that used a DATA step to read a file to extract information that the macro then returned as a single value: call symput("&dummy",value); . Whenever the compiled macro was executed, which injected SAS code which was then also compiled and executed, each time. I replaced that macro with a new one that was 100% purely macro code, which opened the file, used fetchobs to retrieve the desired value, and then "returned" the value: &&&&_&dummy; . Now, when executing the compiled macro, it is simply executed; there is not a second compilation step required.

Usage changed from
[pre]
%let userid = %getID;
%global password;
%getPWD(&userid,password);
[/pre]
to
[pre]
%let userid = %getID;
%let password = %getPWD(&userid);
[/pre]

While in this context the perform gain is not important, the principle still applies where it would be signficant.

Straight regular SAS code or straight macro code will always be faster.

The mixing of SAS and macro code should only be used to simplify the writing of SAS code: to provide a single point of maintenance, to modularize the code to reduce redundancy of effort, to parameterize commonly performed coding efforts.

I have found over my years of experience with SAS programming, that the finely intermingled coding efforts illustrated by your example can always be done better with proper use of regular SAS coding features, that the original coder really didn't understand SAS. I have been guilty of that many times myself.

[deleted_user]

Ahhh I see your point exactly now. Lesson learned, thank your for rehashing through this. I think was a matter of falling into a pattern of coding that normally does not have any negative impact (or so i thought). This has definitely changed my outlook on Macro code vs. SAS code. Thanks again for the insight!

[deleted_user]

You can see another example of "pure" macro coding in the "Proc SQL" thread in the SAS Procedures forum. Look near the end at Swade016's code and at my alternative to some of his stuff. My suggestion removes the need/use for/of 3 data steps and the generation of a number of "indexed" macro variables. The example is derived from what I couldn't produce above.

An example of mine that has mixed macro and regular SAS code is in thread "Creating a user-defined HTML-Output" in the ODS and Database Reporting forum. It shows the use of a MACRO to modularize and parameterize a commonly used section of code within a regular SAS program.

[darrylovia]

My guess is that updating the local symbol table takes some time.

But check this out.... I created 18828 macro variables using a plain old "do loop" and it took less than a second.

When I 18828 macro variables via a macro do loop it took almost 7 seconds. Hmmmmm...


14 data _null_;
15 do i=1 to 18828;
16 call symput('var'||strip(put(i,8.)),i);
17 end;
18 run;

NOTE: Numeric values have been converted to character values at the places given by:
(Line):(Column).
16:41
NOTE: DATA statement used (Total process time):
real time 0.29 seconds
cpu time 0.29 seconds


19
20 %macro test;
21 data _null_;
22 %do i=1 %to 18828;
23 call symput(cats('testvar',strip(put(&i.,8.))),strip(put(&i.,8.)));
24 %end;
25 run;
26 %mend test;
27 %test;

NOTE: DATA statement used (Total process time):
real time 6.90 seconds
cpu time 6.90 seconds

[deleted_user]

Very interesting...thank you sir!

[Cynthia_sas]

Hi:
Just an additional thought or two.

The original code had this -- which was building the PUT statement and then, at execution time, doing I/O (executing the PUT statement).
[pre]
data _null_;
file devnull;
put
%do i=1 %to 18828;
"&&testvar&i."
%end;
;
stop;
run;
[/pre]

So, if the CREATION alone took 7 seconds, as shown in the above log, that means part of your original 18 seconds was taken up 1) resolving that many macro variables and 2) doing file I/O in writing the resolved variable values to FILE DEVNULL operating system file.

To me, this means that you might want to benchmark the macro creation step separate from the macro usage step with the PUT statement on your system. You can easily do this by making 1 macro program for creation and 1 macro program with your DATA _NULL_/FILE DEVNULL step -- and then running them separately to see the time involved in each step separately.

cynthia

[WarrenR_SAS]

The key reason that the DATA step with %DO runs so slowly is that the %DO loop is generating 18,828 separate CALL SYMPUT statements, all of which need to be compiled before the step is run. My tests indicate that the DATA step required more than 36MB (yes, that's MEGAbytes) of memory to handle that many statements.

data _null_;
%do i=1 %to 18828;
call symput(cats('testvar',&i.),&i.);
%end;
run;

The equivalent DATA step with DO instead of %DO needed to compile only one CALL SYMPUT statement and took less than 300k of memory.