topic Re: Macro String getting too long in SAS Programming

Macro String getting too long

david27 — Mon, 10 Aug 2020 21:52:33 GMT

Hello,

I have this:

/*Have*/
%let output=(n1h2_output*n1h2-b3) + (n2h2_output*n2h2-b3);
%let n1h2=(n1h1_n1h2*n1h1-b2) + (n2h1_n1h2*n2h1-b2);
%let n2h2=(n1h1_n2h2*n1h1-b2) + (n2h1_n2h2*n2h1-b2);
%let n1h1=(a_n1h1*a-b1);
%let n2h1=(a_n2h1*a-b1);

%do h=2 %to 1 %by -1;
  %do n=1 %to 2;
    %let output=%sysfunc(tranwrd(&output.,*n&n.h&h.-,*(&&n&n.h&h..)-));
  %end;
%end;

The problem is, the macro string "output" gets too long.

I provided a short example here but for me the macro string goes longer than 65K characters.

Can someone please advise me to code in more efficient/better way?

Thank You

Re: Macro String getting too long

Oligolas — Mon, 10 Aug 2020 14:15:51 GMT

Hi,

you could split the variable or write the code to a file

I'll recommend you to use a dataset to store the results rather than the formulas in macro variables.

Re: Macro String getting too long

Shmuel — Mon, 10 Aug 2020 14:19:44 GMT

It seems that you have entered an endless loop, assigning OUTPUT using loop with &OUTPUT:

%let output=(n1h2_output*n1h2-b3) + (n2h2_output*n2h2-b3);
   /*======*/
%let n1h2=(n1h1_n1h2*n1h1-b2) + (n2h1_n1h2*n2h1-b2);
%let n2h2=(n1h1_n2h2*n1h1-b2) + (n2h1_n2h2*n2h1-b2);
%let n1h1=(a_n1h1*a-b1);
%let n2h1=(a_n2h1*a-b1);
%do h=2 %to 1 %by -1;
  %do n=1 %to 2;
    %let output=%sysfunc(tranwrd(&output.,*n&n.h&h.-,*(&&n&n.h&h..)-));
       /*=====*/               /*======*/
  %end;
%end;

Beyond, it is probably a part of a macro program.

What are you trying to do?

Re: Macro String getting too long

david27 — Mon, 10 Aug 2020 14:22:20 GMT

It basically helps me create a big formula. Longer than 65K.

Yes i am using output from previous output. But its not infinite loop it has boundaries.

Re: Macro String getting too long

Shmuel — Mon, 10 Aug 2020 15:15:39 GMT

try to replace line:

%let output=%sysfunc(tranwrd(&output.,*n&n.h&h.-,*(&&n&n.h&h..)-));

with:

%let output=%sysfunc(tranwrd(strip(&output.),*n&n.h&h.-,*(&&n&n.h&h..)-));

If it doesn't help replace the double %DO loop with a datastep, writing the formula to file

(like formula.sas) and then do %include "<path>/formula.sas"; in your program.

next may be a help to create the formula without the loops:

73         %let n1h2 = A;
 74         %let n2h2 = B;
 75         %let n1h1 = C;
 76         %let n2h1 = D;
 77         %macro m;
 78         %do h=2 %to 1 %by -1;
 79           %do n=1 %to 2;
 80             %put *n&n.h&h.- , *(&&n&n.h&h..)-));
 81           %end;
 82         %end;
 83         %mend;
 84         %m;
 *n1h2- , *(A)-))
 *n2h2- , *(B)-))
 *n1h1- , *(C)-))
 *n2h1- , *(D)-))

enter the right values instead A B C D into TRANWRD function and STRIP(each_part);

you need just 4 lines without loops.

Re: Macro String getting too long

Shmuel — Mon, 10 Aug 2020 15:51:22 GMT

My first option solves the problem:

72         
 73         %macro m;
 74         %let output=(n1h2_output*n1h2-b3) + (n2h2_output*n2h2-b3);
 75         %let n1h2=(n1h1_n1h2*n1h1-b2) + (n2h1_n1h2*n2h1-b2);
 76         %let n2h2=(n1h1_n2h2*n1h1-b2) + (n2h1_n2h2*n2h1-b2);
 77         %let n1h1=(a_n1h1*a-b1);
 78         %let n2h1=(a_n2h1*a-b1);
 79         %do h=2 %to 1 %by -1;
 80           %do n=1 %to 2;
 81             %let output=%sysfunc(tranwrd(strip(&output.),*n&n.h&h.-,*(&&n&n.h&h..)-));
 82           %end;
 83         %end;
 84         %mend;
 85         %m;
 86         %put OUTPUT=&output;
 OUTPUT=strip(strip(strip(strip((n1h2_output*((n1h1_n1h2*((a_n1h1*a-b1))-b2) + (n2h1_n1h2*((a_n2h1*a-b1))-b2))-b3) + 
 (n2h2_output*((n1h1_n2h2*((a_n1h1*a-b1))-b2) + (n2h1_n2h2*((a_n2h1*a-b1))-b2))-b3)))))
 87

Re: Macro String getting too long

david27 — Mon, 10 Aug 2020 16:29:53 GMT

Thank You @Shmuel for helping me here.

But the solution you provided is quite similar to what I have(except for the strip function you used.)

The solution that you are suggesting will not work for long strings.

The macro "H" can start from 10 and the macro "N" could go upto 20. As a result the string gets too long.

I think we have to find a way without creating output in an iterative fashion and put it out in an external file, which I can later do an include upon(something like formula.sas as per your suggestion)

Re: Macro String getting too long

Shmuel — Mon, 10 Aug 2020 19:22:05 GMT

You iterate just 2X2=4 times by loops.

Check the length after each iteration by

%let len = %length(&output); %put LAN=&len;

and check log, does it realy flows out of max possible length;

Re: Macro String getting too long

Tom — Mon, 10 Aug 2020 22:32:03 GMT

So instead of changing the references to NnHh to the value of &NnHh just change it to a reference instead. Then your macro variables will only grow by one character per reference. Note that you need to change the references in ALL of the macro variables, not just in OUTPUT macro variable.

%let output=(n1h2_output*n1h2-b3) + (n2h2_output*n2h2-b3);
%let n1h2=(n1h1_n1h2*n1h1-b2) + (n2h1_n1h2*n2h1-b2);
%let n2h2=(n1h1_n2h2*n1h1-b2) + (n2h1_n2h2*n2h1-b2);
%let n1h1=(a_n1h1*a-b1);
%let n2h1=(a_n2h1*a-b1);

data _null_ ;
  length mvar $32 ;
  call symputx('output',tranwrd(symget('output'),'*n','*&n'));
  do h=2 to 1 by -1;
    do n=1 to 2;
      mvar=cats('n',n,'h',h);
      call symputx(mvar,tranwrd(symget(mvar),'*n','*&n'));
    end;
  end;
run;

%put N1H2=%superq(n1h2);
%put N2H2=%superq(n2h2);
%put N1H1=%superq(n1h1);
%put N2H1=%superq(n2h1);
%put OUTPUT=%superq(output);


%put OUTPUT=&output;

Compare the raw and evaluated versions and notice the difference in length.

N1H2=(n1h1_n1h2*&n1h1-b2) + (n2h1_n1h2*&n2h1-b2)

N2H2=(n1h1_n2h2*&n1h1-b2) + (n2h1_n2h2*&n2h1-b2)

N1H1=(a_n1h1*a-b1)

N2H1=(a_n2h1*a-b1)

OUTPUT=(n1h2_output*&n1h2-b3) + (n2h2_output*&n2h2-b3)

OUTPUT=(n1h2_output*(n1h1_n1h2*(a_n1h1*a-b1)-b2) + (n2h1_n1h2*(a_n2h1*a-b1)-b2)-b3) + (n2h2_output*(n1h1_n2h2*(a_n1h1*a-b1)-b2) + (n2h1_n2h2*(a_n2h1*a-b1)-b2)-b3)

Note you might need to use regular expression instead of TRANWRD() to find and replace the variable references.

Re: Macro String getting too long

Tom — Mon, 10 Aug 2020 23:18:03 GMT

Here is RegEx version. Replaces patterns that have *NdHd- with *(&NdHd)-.

data _null_ ;
  length mvar $32 ;
  rexid = prxparse('s/\*(n\d+h\d+)-/\*(&$1)-/i');
  do h=2 to 1 by -1;
    do n=1 to 2;
      mvar=cats('n',n,'h',h);
      call symputx(mvar,prxchange(rexid,-1,symget(mvar)));
    end;
  end;
  call symputx('output',prxchange(rexid,-1,symget('output')));
run;

Re: Macro String getting too long

david27 — Tue, 11 Aug 2020 13:16:20 GMT

Thank You very much @Tom @Shmuel @Oligolas for helping me here.

But I think, Shell Scripting is the solution here:

The reason for that is, eventually the macro string output is going to be longer than 65K

data _null_ ;
file "/code/location/on/server/formula.sas";
put  "output=" "&output.";
run;

%do h=10 %to 1 %by -1;
%do n=1 %to 20;
	systask 
		command "sed -i 's/*n&n.h&h.-/*(&&n&n.h&h..)-/g' /code/location/on/server/formula.sas"
		wait
		status=status
		shell;
	%put &=status.;
%end;
%end;

Re: Macro String getting too long

Tom — Tue, 11 Aug 2020 13:32:31 GMT

Note that using the nested macro variable references I showed the result of evaluating &OUTPUT can be longer than 64K even when the actual length of the macro variable value is not.

I don't really see how calling SED is going to help, but then I don't understand what you are trying to do. It looks like you are generating a formula. So you might use the macro variable to place that formula into some SAS code like this:

data want;
  set have ;
  new_var = &output ;
run;

Do you really expect to execute a formula that is longer than 64K?

Re: Macro String getting too long

Shmuel — Tue, 11 Aug 2020 14:28:21 GMT

May be the solution is not by creating a huge macro variable but by splitting it into a set of formulas,

creating something like:

v_tmp1 = &formula1;
v_tmp2 = v_tmp1 * &formula2;
v_tmp3 = v_tmp2 * &formula3;
...etc...