Macro variable resolution bug with % sign

[Quentin]

Woke up this morning to an interesting post on stack overflow reporting a macro resolution bug.

https://stackoverflow.com/questions/76997015/what-happend-when-a-macro-variable-with-percent-symbol-...

Coding:

options symbolgen ;
%let x=% ;
%let y=B ;
%put &x A&y ;

should return % AB but it returns % AB AB and symbolgen even shows that Y resolves twice:

1    options symbolgen ;
2
3    %let x=% ;
4    %let y=B ;
5
6    %put &x A&y ;
SYMBOLGEN:  Macro variable X resolves to %
SYMBOLGEN:  Macro variable Y resolves to B
SYMBOLGEN:  Macro variable Y resolves to B
% AB  AB

I thought "that's a weird bug in the %PUT statement" and then got more concerned when I saw that it really is a macro resolution bug, e.g.:

8    data _null_ ;
9     x ="&x A&y" ;
SYMBOLGEN:  Macro variable X resolves to %
SYMBOLGEN:  Macro variable Y resolves to B
SYMBOLGEN:  Macro variable Y resolves to B
10    put x= ;
11   run ;
x=% AB AB

Anyone seen this before or better yet, have an explanation for how this could be tripping up the macro processor?

[Tom]

You can try raising a SAS support ticket, but since this behavior has existed since at least SAS 9.2 I am not sure they will fix it.

It seems be triggered only when SAS is confused about whether to treat that as one token or two.  For example if you replace the space between the two with a + or another character that would start a new token the double generation does not happen.

You can also avoid it be adding macro quoting the the macro variable that has the bare % in it.

[Quentin]

Agree it's the bare % that is somehow leading to the problem.  Even adding trailing spaces to the value in the macro var doesn't prevent the problem:

124  data _null_ ;
125    call symput("x","Foo %  ") ;
126    call symput("y","B") ;
127  run ;


SYMBOLGEN:  Macro variable X resolves to Foo %
128
129  %put >>&x<< ;
>>Foo %  <<
130  %put &x A&y ;
SYMBOLGEN:  Macro variable X resolves to Foo %
SYMBOLGEN:  Macro variable Y resolves to B
SYMBOLGEN:  Macro variable Y resolves to B
Foo %     AB  AB

Agree it's not likely to be fixed.  But still curious about what could cause this behavior.  Even with various macro resolution problems, I don't think I've ever seen the macro processor return too much text.  I guess maybe it's a word scanner bug.  Where the word scanner sends &y off to the macro processor to be resolved, then builds the token AB, then for some reason does it a second time.

[Quentin]

Came back to this at the end of the day and stumbled across an oddity.

I thought maybe if I used a number instead of A, problem wouldn't happen, because %1 cannot be a macro call so I thought it might tokenize properly.  I tried:

%let x=% ;
%let y=B ;
%put &x 1&y ;

And it did repeat. So that theory was wrong.

Then I noticed that:

%let x=% ;
%let y=B ;
%put &x 1FOO&y ; *does not repeat! ;

does not repeat. 

Then I noticed that if the characters before &y are a valid hex number, it will repeat.  But if they start with a digit but the string prefix string is not a valid hex number, it doesn't repeat:

%let x=% ;
%let y=B ;

%put &x  1B&y   ; *hex number duplicates ;
%put &x  1C&y   ; *hex number duplicates ;
%put &x  1D&y   ; *hex number duplicates ;
%put &x  1E&y   ; *hex number duplicates ;
%put &x  1F&y   ; *hex number duplicates ;

%put &x  1G&y   ; *does not duplicate !!! ;
%put &x  1H&y   ; *does not duplicate !!! ;

%put &x  1CG&y  ; *does not duplicate !!! ;
%put &x  1DG&y  ; *does not duplicate !!! ;
%put &x  1EG&y  ; *does not duplicate !!! ;
%put &x  1FG&y  ; *does not duplicate !!! ;

Not sure what to infer from that, if anything.  But thinking about hex codes made me think about quoting characters and automatic unquoting.... I don't know.

[ballardw]

I think it is more related to the bare %.

1    options symbolgen ;
2    %let x=%nrstr(%%) ;
3    %let y=B ;
4    %put &x A&y ;
SYMBOLGEN:  Macro variable X resolves to %
SYMBOLGEN:  Some characters in the above value which were subject to macro
            quoting have been unquoted for printing.
SYMBOLGEN:  Macro variable Y resolves to B
% AB

Note that %nrstr(%); is one of those things that creates and unstable macro situation and you only see the error about an unfound closing parentheses when shutting down SAS, at least with a relatively "clean" session just testing this code. Use of %str(%%) does the same behavior as observed.

[Tom]

%SUPERQ() seems to protect against it.

34   %let x=%;
35   %let y=YSTRING;
36   %put No quoting &x A&y;
No quoting % AYSTRING  AYSTRING
37   %let x=%superq(x);
38   %put SUPERQ &x A&y;
SUPERQ % AYSTRING

[Tom]

Different types of macro quoting get stored differently into the macro variables.

135  %let x1=%;
136  %let x2=%superq(x1);
137  %let x3=%bquote(&x1);
138  %let x4=%quote(&x2);
139  %let x5=%nrstr(%%);
140  %let x6=%str(%%);
141
142  data test;
143    set sashelp.vmacro;
144    where name=:'X';
145    len=length(value);
146    value2 = putc(value,cats('$hex',2*len,'.'));
147    value=symget(name);
148    put name= len= value=  value2=;
149
150  run;

name=X1 len=1 value=% value2=25
name=X2 len=3 value=% value2=061008
name=X3 len=3 value=% value2=042508
name=X4 len=5 value=% value2=0306100808
name=X5 len=3 value=% value2=011002
name=X6 len=3 value=% value2=012502
NOTE: There were 6 observations read from the data set SASHELP.VMACRO.
      WHERE name=:'X';
NOTE: The data set WORK.TEST has 6 observations and 6 variables.
NOTE: DATA statement used (Total process time):
      real time           0.00 seconds
      cpu time            0.00 second

[Ksharp]

[whymath · Accepted solution]

Thank @Quentin  for posting this question to community. I have asked SAS IT support and it turns out this is a known bug but not fixed yet. If you see this webpage, you can find a workaround:

This bug fixing process have a more severe impact on the processing of tokens and the change was backed out since there is a fairly easy workaround available in documentation.