Interesting word scanner bug when tokenizing %MacroInvocation/*%MacroInvocation*/

[Quentin]

Hi All,

Below is some test code, that when I run in Windows 9.3 and 9.4  DM SAS or batch submit, shows what looks like an interesting work scanner bug.

This code:

  %outer()/*%iShouldNotRun()  but I do */

Results in the macro %iShouldNotRun being invoked, even though it is inside a comment block.  Interestingly, the text BUT I DO is not executed, suggesting perhaps is a macro processor bug.  But then, I thought the macro processor should never even see text inside of /* */ comments, so I think I'm blaming the word scanner/tokenizer/SAS supervisor(?).

Also interestingly, the code:

  %inner()/*%iShouldNotRun()  and I do not */

does not result in %iShouldNotRun() being invoked. It seems to only occur when the invoked macro calls another macro (Even that is not accurate, some calls to macro functions cause the problem, some do not, and some calls to autocall macros do not).

While I accept that it is odd to have no white space between the closing parenthesis and the /*  I don't think it violates any syntax rules.

I throw it out to the group in case someone with better knowledge of tokenizers/compilers than me can provide some insight as to what might be going wrong, and what logical construct is to blame.

Code/Log:

%macro outer(dummy);
  %put &sysmacroname is running;
  %inner()
%mend outer;

%macro inner(dummy);
  %put &sysmacroname is running;  
%mend inner;

%macro iShouldNotRun(dummy);
  %put &sysmacroname is running;  
%mend iShouldNotRun;

%outer()/*%iShouldNotRun()  but I do */
%outer()/* %iShouldNotRun() and I do not */
%outer() /*%iShouldNotRun() and I do not */
%inner()/*%iShouldNotRun()  and I do not */

My Log:
1          %macro outer(dummy);
2            %put &sysmacroname is running;
3            %inner()
4          %mend outer;
5          
6          %macro inner(dummy);
7            %put &sysmacroname is running;
8          %mend inner;
9          
10         %macro iShouldNotRun(dummy);
11           %put &sysmacroname is running;
12         %mend iShouldNotRun;
13         
14         %outer()/*%iShouldNotRun()  but I do */
OUTER is running
INNER is running
ISHOULDNOTRUN is running
15         %outer()/* %iShouldNotRun() and I do not */
OUTER is running
INNER is running
16         %outer() /*%iShouldNotRun() and I do not */
OUTER is running
INNER is running
17         %inner()/*%iShouldNotRun()  and I do not */
INNER is running

[ballardw]

Not the behavior in 9.2.3

14   %outer()

OUTER is running

INNER is running

15   %outer()

OUTER is running

INNER is running

16   %outer()

OUTER is running

INNER is running

17   %inner()

INNER is running

[Quentin]

Interesting that it's not the behavior on 9.2.  I and others have confirmed it on 9.3 and 9.4 Windows.  Perhaps it's new.

[RW9]

Not sure its a bug.  I remember there being something about SAS reading the next statement if there is no delimiter found.  To me though, it looks more like your actually trying to break it.  Missing semicolons in various places for instance.  Generally speaking I wouldn't put comments on the code line either, most of the time its hard to read, and you can't use the cntrl+/ to turn comments on and off.  So to fix it, put delimiters in the correct place:

%outer();/*%iShouldNotRun()  but I do */ /* No longer will I run */

[Quentin]

Macro invocations do not need (or want) semicolons to end them (even if the SAS syntax highlighters don't know that. : )  The macro invocation ends with the closing parenthesis.

Semicolons after a macro invocation will add an extra SAS semicolon to your code, and do not allow you to use function-style macros, e.g.:

  %put At cookouts I eat %lowcase(HoT)dogs.;

That said, I agree that lack of white space between the closing parenthesis and start of comment is unusual, and would not win any style points.  But still seems like a bug to me.

[RW9]

Hi Quentin,

Yes, when talking about inline macros, then you wouldn't want the semicolon otherwise it would produce invalid code.  So in your example %inner() is ok, however when the macro is not inline, then I would still suggest that its good practice to put a semicolon.  As mentioned, in the example:

%outer();/*%iShouldNotRun()  but I do */ /* No longer will I run */

Putting the semicolon stops the problem (in some instances white space is the delimiter).  Same as run; not being needed, or newlines etc.

Can't test < 9.3, but your right it is on 9.3 and 9.4.  I would suspect it is something to do with the macro's only having macro syntax -> i.e. they don't actually do anything, for example this works fine:

%macro check (v=);

  %convert (v=&v.)

%mend check;

%macro convert (v=);

  if &v.="M" then result="male"

%mend convert;

data test;

  set sashelp.class;

  %check(v=sex)/*%ishouldnotrun()*/;

run;

And strangely enough adding a further layer on top also seems to work fine:

%macro further_out(dummy);

  %put &sysmacroname is running;

  %outer()

%mend;

%macro outer(dummy);

  %put &sysmacroname is running;

  %inner()

%mend outer;

%macro inner(dummy);

  %put &sysmacroname is running; 

%mend inner;

%macro iShouldNotRun(dummy);

  %put &sysmacroname is running; 

%mend iShouldNotRun;

%further_out()/*%shouldnotrun()*/

[Quentin]

The fact that %further_out works fine just blew my mind (and any half-baked theories I was developing to try to understand this).  I even tried adding %even_further_out to see if the problem would magically recur, but happily it did not.  Agree, the problem seems limited to macro that have only macro code.  I think I'll send it in to tech support.

Thanks All!

[Ksharp]

I can repeat your problem . It appear SAS eat  '  /*  '  , in other words,  SAS don't take ' ) ' as the end character of macro .

Very interesting !.

%outer()*%iShouldNotRun()  but I do ;

%outer()* %iShouldNotRun() and I do not */

206  %outer()*%iShouldNotRun()  but I do ;

OUTER is running

INNER is running

ISHOULDNOTRUN is running

207  %outer()* %iShouldNotRun() and I do not */

OUTER is running

INNER is running

Another Funny thing is :

302  %outer()***%iShouldNotRun()  but I do ;

OUTER is running

INNER is running

303  %outer()* %iShouldNotRun() and I do not */

OUTER is running

INNER is running

Message was edited by: xia keshan

[jakarman]

It is not a bug it is defined to work that way. SAS(R) 9.4 Macro Language: Reference, Third Edition
The macro is replacing all text when it passed that to the sas-compiler.  Wihtin the SAS compiler * ; and /* * /  are comments.
But as the macro is another language and not the compiler it should replace all that. I you want to use comment in the macro language use %* ; SAS(R) 9.4 Macro Language: Reference, Third Edition/ What a token is is defined at SAS(R) 9.4 Macro Language: Reference, Third Edition.

More interesting the semicolon:
- sas statements are ending with a ;
- macro statements are ending with a ;
- macro invocations are NOT ending with a ;
  macro variables are ended with a  dot .  and the macro call will run after ( ) as end of call or next boundary.  

[Quentin]

I don't think it's defined that way at all : )

Note in the page you linked (thanks):

From the time the word scanner triggers the macro processor until that macro processor action is complete, the macro processor controls all activity. When the macro processor is active, no activity occurs in the word scanner or the DATA step compiler. When the macro processor is finished, the word scanner reads the next token (the DATA keyword in this example) and sends it to the compiler. The word scanner triggers the compiler, which begins to pull tokens from the top of the queue, as shown in the following figure.

If I code:

  %outer()/*%me is a macro that should not be called*/

So when the word scanner sees the macro trigger (%o in this case) it starts sending tokens to the macro processor.  So %outer() is sent to the macro processor, and it does its work.  Now the macro processor is finished.  Control goes back to the word scanner, it should read the next token, /* , and see that it is the start of a comment block.

Cant see any explanation why

  %outer()/*%me is a macro that should not be called*/

would invoke %me, but

  %outer() /*%me is a macro that should not be called*/

would not and

  %outer()/* %me*/

would not

In al cases, the word scanner should see a /* as the start of comment block, and should not execute the macro.

[jakarman]

Quentin, We make assumptions on how it work or should work. Guessing is bad for a reliable approach.  

1/ you are assuming the comment block is recognized by /*,  /* is not a token instead is are two of them the / and *

   With the SAS coding they can occur separately   the * for starting comment  and the / for an option.  The %copy call is supporting a / options after the () calling.

2/ you are assuming within  a comment block the macro processor is not active. It is nowhere documented that it doesn't. I did have bad experienced with comment block causing the macro processing to fail.  Either is changed after that or ....? .  Never mind SAS didn't document those kind of exceptions of recognized language than it should work.  That is why I did the opposite opinion. I see now SAS(R) 9.4 Macro Language: Reference, Third Edition . My bad experiences are the same type of errors they should not have happened.
  

3/ that / (possible an option) after closing brackets could be the one together with the nested macro-calls.  the %COPY statement I supporting that. The tokenizer could be wrong in that situation.

I remember having seen the question of not having that magic-string at the end of code (Eguide Studio). It masks things that are wrong when code is use different.

[Quentin]

Okay, I can accept that /* are two tokens.

But from the page you linked, you accept that the macro calls inside of /* */ should not be executed, right, so now agree it's a bug?

Only macro comment statements and SAS comments of the form /*commentary*/ in macro definitions or open code might be used to hide macro statements from processing by the macro facility. 
...
SAS comments in the form 
/*commentary*/
are not tokenized, but are processed as a string of characters. These comments can appear anywhere a single blank can appear and can contain semicolons or unmatched quotation marks. SAS comments in the form 

So this example is exactly that, a /*comment*/ appearing where a blank could appear.

[jakarman]

Yep Quentin, I agree it is a bug.   ( I believe there are some more minor ones and those are there for a long time)

[Quentin]

Thanks much Jaap as always!  I always feel more confident when I can agree with you. : )

[Ksharp]

Yes. I also believe it is a bug of macro compiler . It is very funny thing. It seems will eat the first /* or * after ) , but if there is a some other character between ) and /* or * , macro compiler will behavior normally . Which remind us never put ) and /* or * together one by one ,must have some blanks between them. Very Very interesting ! Check the following examples:

1)

375  %outer()%iShouldNotRun() 

OUTER is running

INNER is running

ISHOULDNOTRUN is running

2)

391  %outer()*%iShouldNotRun()

OUTER is running

INNER is running

ISHOULDNOTRUN is running

3)

407  %outer()**%iShouldNotRun()

OUTER is running

INNER is running

ISHOULDNOTRUN is running

4)

423  %outer()***%iShouldNotRun()

OUTER is running

INNER is running

5)

471  %outer()/*%iShouldNotRun()

OUTER is running

INNER is running

ISHOULDNOTRUN is running

6)

487  %outer()/*/*%iShouldNotRun()

OUTER is running

INNER is running

7)

503  %outer()/**%iShouldNotRun()

OUTER is running

INNER is running

Xia Keshan