How does %sysfunc(countw(... get resolved in a %do loop

[Lukkul]

Dear Sas Community,

I'm having troubles understanding why this code, with %str(;) inside of a sysfunc call in a loop is not working:

%macro split_terms;
%let terms = %str(This is a term; This is another term, with a comma; And this is a third term);
%do i = 1 %to %sysfunc(countw(&terms, %str(;)));
%put %scan(&terms, &i, %str(;));
%end;
%mend;
%split_terms;

But when in sysfunc, I put quotes around semicolon and remove %str like below, it works correctly:

%macro split_terms;
%let terms = %str(This is a term; This is another term, with a comma; And this is a third term);
%do i = 1 %to %sysfunc(countw(&terms, ';'));
%put %scan(&terms, &i, %str(;));
%end;
%mend;
%split_terms;

The other way around it is to put the resolution of %sysfunc(countw(&terms, %str(;))) into macro variable like:

%macro split_terms;
    %let terms = %str(This is a term; This is another term, with a comma; And this is a third term);
    %let nterms = %sysfunc(countw(&terms, %str(;)));
    %do i = 1 %to &nterms.;
        %put %scan(&terms, &i, %str(;));
    %end;
%mend;
%split_terms;

In other statements, like %if, there seems to be no difference in behavior:

%let terms = %str(This is a term; This is another term, with a comma; And this is a third term);
%if %sysfunc(countw(&terms, %str(;))) eq %sysfunc(countw(&terms, ';')) %then %do;
    %put ---> EQUAL <---;
%END;

24 %put ---> EQUAL <---;
---> EQUAL <---
25 %END;

Is there a difference of behavior when such %sysfunc(countw... statement is used in a loop?

[yabwon]

It's look like a but when handling the ";" character by the %sysfunc(countw(...))  combo in the "%to" part of the %do-loop.

When you put the separator as a macrovariable:

%macro split_terms();
%let terms = %str(A; B,C; D);

%let sep=%str(;);

%do i = 1 %to %sysfunc(countw( &terms., &sep. ));
  %put &=i.;
  %put %scan(&terms, &i, %str(;) );
%end;
%mend;
%split_terms()

you will get what you want:

[...]
10   %mend;
11   %split_terms()
I=1
A
I=2
B,C
I=3
D

Also when you ignore the the third argument it will run ok:

%macro iterm(lst);
 %let finish=%sysfunc(countw(&lst));
  %do i = 1 %to &finish;
   %put %scan(&lst,&i);
  %end;
%mend iterm;
%iterm(a; c; e)

log is:

[...]
6    %mend iterm;
7    %iterm(a; c; e)
a
c
e

BTW.  that example is copy-pasted from the SAS doc, I only added ";" to the list.

I would report that to the SAS Tech support and ask for investigation.

Bart

[yabwon]

And it's not a matter of comma, since the "&sep." trick cuts the data ok.

And adding execution time quoting functions wont help too:

%macro split_terms();
%let terms = %str(A; B,C; D);

%do i = 1 %to %sysfunc(countw( %superq(terms), %str(;) ));
  %put &=i.;
  %put %scan(%superq(terms), &i, %str(;) );
%end;
%mend;
%split_terms()

log says:

1    %macro split_terms();
2    %let terms = %str(A; B,C; D);
3
4    %do i = 1 %to %sysfunc(countw( %superq(terms), %str(;) ));
5      %put &=i.;
6      %put %scan(%superq(terms), &i, %str(;) );
7    %end;
8    %mend;
9    %split_terms()
ERROR: Expected close parenthesis after macro function invocation not found.
NOTE: One or more missing close parentheses have been supplied for the %COUNTW function.
ERROR: Expected close parenthesis after macro function invocation not found.
ERROR: %EVAL function has no expression to evaluate, or %IF statement has no condition.
ERROR: The %TO value of the %DO I loop is invalid.
ERROR: The macro SPLIT_TERMS will stop executing.

it looks like somewhere under the hood that semicolon is breaking something...

Bart

[yabwon]

BTW the approach with quotes around semicolon is not good idea too. If by chance your text would contain commas they would be used as separators:

%macro split_terms();
%let terms = %str(A; B,'C',D; E);

%do i = 1 %to %sysfunc(countw( &terms., ';' ));
  %put &=i.;
  %put %scan(&terms., &i, ';' );
%end;
%mend;
%split_terms()

Log is:

[...]
8    %mend;
9    %split_terms()
I=1
A
I=2
B,
I=3
C
I=4
,D
I=5
E

Bart

[Lukkul]

Thanks for your inputs Bart ♥️
The approach with single quotes does the trick, but only if semicolon is quoted in sysfunc call, it still has to be wrapped in %str in scan function like:

%macro split_terms;
%let terms = %str(A; B,'C',D; E);
%do i = 1 %to %sysfunc(countw(&terms, ';'));
%put %scan(&terms, &i, %str(;));
%end;
%mend;
%split_terms;

Then the output is 

A
B,'C',D
E

Which is just bananas... 😃

[yabwon]

Add: 

%put &=i.;

inside the loop you will see it works wrong.

Bart

[Quentin]

Yes, please report to tech support.

I thought this was about %SYSFUNC doing something weird.  But after testing, it happens even without %SYFUNC, so I think it's something happening in the %DO statement itself.

When I submit:

%macro try();
%do i =1 %to %length(%str(abc;def));
  %put &=i ;
%end;
%mend;
%try();

I get a macro execution-time error:

718  %macro try();
719  %do i =1 %to %length(%str(abc;def));
720    %put &=i ;
721  %end;
722  %mend;
NOTE: The macro TRY completed compilation without errors.
      11 instructions 180 bytes.
723  %try();
ERROR: Expected close parenthesis after macro function invocation not found.
NOTE: One or more missing close parentheses have been supplied for the %LENGTH function.
NOTE: Line generated by the invoked macro "TRY".
1     def));
      ---
      180
MPRINT(TRY):   def));

ERROR 180-322: Statement is not valid or it is used out of proper order.

I=1
NOTE: Line generated by the invoked macro "TRY".
3    def));
     ---
     180
MPRINT(TRY):  def));

ERROR 180-322: Statement is not valid or it is used out of proper order.

I=2
NOTE: Line generated by the invoked macro "TRY".
5    def));
     ---
     180
MPRINT(TRY):  def));

ERROR 180-322: Statement is not valid or it is used out of proper order.

I=3

And it's interesting that the macro does actually execute 3 times.  So clearly that semicolon was unmasked somehow.

And if you take your original code and move the %sysfunc(countw()) before the %TO, you get an error during macro compilation, indicating that the semicolon was seen and broke compilation.

I think in both cases, it's  an error that happens during macro compilation.

%macro split_terms;
%do i = %sysfunc(countw(foo, %str(;))) %to 1 ; 
%end;
%mend;

724  %macro split_terms;
725  %do i = %sysfunc(countw(foo, %str(;))) %to 1 ;
ERROR: Expected %TO not found in %DO statement.
ERROR: A dummy macro will be compiled.
ERROR: Improper use of macro reserved word TO.
726  %end;
727  %mend;
NOTE: The macro SPLIT_TERMS completed compilation with errors.
      0 instructions 0 bytes.

The %DO statement is doing stuff behind the scenes, like it has an implied %EVAL.  But I couldn't replicate the problem with an %IF statement which also has an implied eval.

[yabwon]

Also the following "blows up":

resetline;
%macro test(a);
  %scan(%superq(a),1)
%mend;


%macro split_terms();
%do i = 1 %to %test(%str(123;456));
  %put &=i.;
%end;
%mend;
%split_terms()

but this (with an extra macrovariable) works like a charm:

resetline;
%macro test(a);
  %scan(%superq(a),1)
%mend;


%macro split_terms();
%let x = %test(%str(123;456));
%do i = 1 %to &x.;
  %put &=i.;
%end;
%mend;
%split_terms()

I agree with @Quentin you should report it. 

If SAS does not decide to fix that, they can at least update the definition in the documentation: https://documentation.sas.com/doc/en/pgmsascdc/9.4_3.5/mcrolref/p0ri72c3ud2fdtn1qzs2q9vvdiwk.htm

because now it states:

start

specifies an integer (or a macro expression that generates an integer) to use as the initial value of macro-variable. Start and stop control the number of times the statements between the iterative %DO statement and the %END statement are processed. For the first iteration, macro-variable is equal to start. As processing continues, the value of macro-variable changes by the value of increment until the value of macro-variable is outside the range of integers specified by start and stop.

%TO stop

specifies an integer (or a macro expression that generates an integer) to use as the final macro-variable iteration value. Iteration stops if the value of macro-variable exceeds this value.

Bart

[Quentin]

Here's an interesting one:

%macro try2();
%do i =1 %to %length(%str(abc;
  %put &=i ;
%end;
%mend;

%try2();

That macro happily compiles, so that semicolon is seen by the macro compiler and ends the %do statement.

And then when it executes, you do get an error because the %length function is missing the closing ) , but SAS adds it in, and the macro 'works':

1    %macro try2();
2    %do i =1 %to %length(%str(abc;
3      %put &=i ;
4    %end;
5    %mend;
NOTE: The macro TRY2 completed compilation without errors.
      9 instructions 144 bytes.
6
7    %try2();
ERROR: Expected close parenthesis after macro function invocation not found.
NOTE: One or more missing close parentheses have been supplied for the %LENGTH function.
I=1
I=2
I=3

[yabwon]

I think compilation of macro without closing bracket is ok. The same way this:

%macro test(a);
  %length(%str(&a.
%mend;

compiles (aka stores macro code in a catalog), or this:

%macro test(a);
  %noSuchMacro(()
%mend;

because the process "only" stores the macro code. And both cases it is treated just as a "text to be generated during execution".

And only in the execution phase they both blow up.

Bart

[Quentin]

But I think for the %DO statement it's different.  I think it really does compile during macro compilation time.  Differently than a %length statement "compiles".  I think. : )

[Quentin]

Reading again through all these examples (thanks @yabwon for adding a bunch!).

I think it's a problem that happens during macro compile time.  Where when the %DO statement compiles it is either unquoting the semicolon or, for some reason, %str(;) is never actually quoting the semicolon.  So the semicolon prematurely ends the %DO statement, and it compiles incorrectly.  Then the everything else, i.e. ))); or whatever, becomes just text.

But I can't think of an explanation.

[Lukkul]

Thank you Bart and Quentin for the discussion 😊

Luckily, the bit with semicolon in a macro variable works correctly, so there is a workaround, but I'll be reporting the issue for sure 

[Quentin]

---

@Lukkul wrote:

Thank you Bart and Quentin for the discussion 😊

Luckily, the bit with semicolon in a macro variable works correctly, so there is a workaround, but I'll be reporting the issue for sure 

---

Yes, @yabwon's first example was I think not only workaround but also explanation.  The fact that:

%do i = 1 %to %sysfunc(countw( &terms., &sep. ));

works suggests it's a problem that happens at compile time.  Essentially, instead of using %str() to mask the semicolon that appears on the %DO statement, Bart avoids having a semicolon in the %DO statement.  So this %DO statement can correctly compile without being ended prematurely by the semicolon.  And it also executes correctly. : )

[Tom]

This bug has existed since at least version 9.2

   8? %macro try;%do i=1 %to %quote(2;3);%put i=&i;%end;%mend;%try;

NOTE: Line generated by the invoked macro "TRY".
8     3);
      -
      180
ERROR 180-322: Statement is not valid or it is used out of proper order.

i=1
NOTE: Line generated by the invoked macro "TRY".
8     3);
      -
      180
ERROR 180-322: Statement is not valid or it is used out of proper order.

i=2

   9?