substrn strange behavior when called via %sysfunc, evaluates first param and ignores macro quoting

[john_floralde]

Our project uses SAS 9.4M7 on Linux.
Edit on formatting: 
Here is the sample code:

%let calcsets=CA DE GP LE;
%let cs= DE;
%let calcSetsPrev=%sysfunc(substrn(%superq(calcsets), 1, %sysfunc(findw(&calcsets,&cs)) -1 ));

Is not returning CA;

Our workaround was to do use normal quotes instead.

%let calcsets=CA DE GP LE;
%let cs= DE;
%let calcSetsPrev=%sysfunc(substrn("&calcsets", 2, %sysfunc(findw(&calcsets,&cs)) -2 ));

Is returning CA.

Anybody else encountered this? For comparison using substrn in data step works fine.

[Patrick · Accepted solution]

I could replicate what you describe under Windows and SAS 9.04.01M7

Looks to me like you found a bug - which is rather rare but very worthwhile raising with SAS Tech Support.

Alternative code that should work:

%let calcSetsPrev=%substr(&calcsets, 1, %eval(%sysfunc(findw(&calcsets,&cs)) -1) );

It appears that the source text string doesn't get properly isolated/quoted. If there is an operator keyword in it then this becomes somehow part of actual logic instead of just being the string. 

I've only observed this with the substrn() function. Things work as expected when using substr().

Result when using substr()

Please let me know if you're not raising this with SAS TS as then I'm going to do it. We all want a bug-free SAS.

[john_floralde]

Interesting. If its not much effort from your side then please feel free to raise it. Many thanks for also testing it in Windows.

[Patrick]

ok - I'm gonna raise this one with SAS TS.

Update: Raised with SAS TS. 

[yabwon]

Substrn() seems to be doing the same it does in data step, i.e. evaluation and type conversion on numeric values.

1    data _null_;
2      Result1=substrn(2 LE 1, 1);
3      Result2=substrn(2 GE 1, 1);
4      Result3=substrn(2 EQ 1, 1);
5
6      Result4=substrn("2" EQ "2", 1);
7      put (_all_) (=/);
8    run;


Result1=0
Result2=1
Result3=0
Result4=1
NOTE: DATA statement used (Total process time):
      real time           0.00 seconds
      cpu time            0.00 seconds


9
10
11   %let a=%sysfunc(substrn(2 GE 1, 1));
12   %put &=a.;
A=1
13   %let b=%sysfunc(substrn(2 LE 1, 1));
14   %put &=b.;
B=0
15   %let c=%sysfunc(substrn(2 EQ 1, 1));
16   %put &=c.;
C=0
17   %let d=%sysfunc(substrn(2 EQ 2, 1));
18   %put &=d.;
D=1

Bart

[Patrick]

@yabwon 

I - sort of - can understand that if in a data step the first parameter isn't passed in as a string in quotes then it gets executed/resolved. I still have a bit of an issue understanding why this happens on macro level as well. 

Given other functions I would have thought one would need to "trigger" such behavior via a 2nd %sysfunc(). 

....If such logical evaluation returning a true/false value ever makes sense for a string function would then be another question.

Consulting the SAS docu (which I should have done from start....):

It appears the SAS docu somehow addresses the challenge. Only.... I haven't found any macro quoting function that prevented the first argument from being evaluated.

[yabwon]

IMHO it's very strange on the macro level.

First I thought it's because in "2 GE 1" string there is "nothing to quote", but it also happens for "2 >= 1" and even for "%STR(2 GE 1)", and the next thing is that quoting functions always add "quotes" around a string, so after all it should be masked.

1    %let a1 = 2 GE 1;
2    %let a=%sysfunc(substrn(%superq(a1), 1));
3    %put &=a1. &=a.;
A1=2 GE 1 A=1
4
5    %let b1 = 2 >= 1;
6    %let b=%sysfunc(substrn(%superq(b1), 1));
7    %put &=b1. &=b.;
B1=2 >= 1 B=1
8
9    %let c1 = %str(2 GE 1);
10   %let c=%sysfunc(substrn(%superq(c1), 1));
11   %put &=c1. &=c.;
C1=2 GE 1 C=1
12
13   %let d1 = %nrstr(2 GE 1);
14   %let d=%Qsysfunc(substrn(%superq(d1), 1));
15   %put &=d1. &=d.;
D1=2 GE 1 D=1

looks like a bug 🙂 but also looks like a feature which allows to evaluate quoted expressions:

1    %let a1 = 2 GE 1;
2    %let a=%sysfunc(substrn(%superq(a1), 1));
3
4    %let x=%sysevalf(%superq(a1)); /* quoted is not evaluated */
ERROR: A character operand was found in the %EVAL function or %IF condition where a numeric
       operand is required. The condition was: 2 GE 1
5
6    %let y=%sysevalf(&a1.);
7    %put &=a1. &=a. &=x. &=y.;
A1=2 GE 1 A=1 X= Y=1

[EDIT:] one more example:

1    %let t = 3;
2    %let a1 = %nrstr(2.0 + 1 + &t.);
3    %let a=%sysfunc(substrn(%superq(a1), 1));
4
5    %let x=%sysevalf(%superq(a1));
ERROR: A character operand was found in the %EVAL function or %IF condition where a numeric
       operand is required. The condition was: 2.0 + 1 + &t.
6
7    %let y=%sysevalf(&a1.);
ERROR: A character operand was found in the %EVAL function or %IF condition where a numeric
       operand is required. The condition was: 2.0 + 1 + &t.
8    %put &=a1. &=a. &=x. &=y.;
A1=2.0 + 1 + &t. A=6 X= Y=

All the best

Bart

[Patrick]

Just as an update:

As @Tom explained the function works as designed. 

I've raised a TS track and it has been accepted that an update to the SAS docu might help to further clarify things given that the "behavior as designed" when using the function within %sysfunc() might not be "intuitive" and as expected.

[Tom]

I doubt it is specific to SUBSTRN() as it is caused by the use of %SYSFUNC() to try and call a function than can use either numeric or character values for an argument.  In that case %SYSFUNC() has to decide which you are passing.  Your string looks like a logical expression to %SYSFUNC() so it is passing the resulting 0/1 result to the SUBSTRN() function.

Here are actual SAS macros that use macro logic and the %SUBSTR() function to mimic the functionality of the SUBSTRN() function in macro code.

https://github.com/sasutils/macros/blob/master/substrn.sas

https://github.com/sasutils/macros/blob/master/qsubstrn.sas

325   %put calcSetsPrev=%substrn(%superq(calcsets), 1, %sysfunc(findw(&calcsets,&cs)) -1 );
calcSetsPrev=CA
326   %put calcSetsPrev=%substrn(&calcsets, 1, %sysfunc(findw(&calcsets,&cs)) -1 );
calcSetsPrev=CA

[Tom]

It is not a "bug" it is a feature of how %SYSFUNC() interacts with functions that can take either numeric or character arguments.  The SUBSTRN() allows numeric or character values for the first argument.

300   data _null_;
301     n=0;
302     str='CA DE GP LE';
303     test1=substrn(str,1,3);
304     test2=substrn(n,1,3);
305     put (_all_) (=);
306   run;

n=0 str=CA DE GP LE test1=CA test2=0

The macro processor when evaluating the call to %SYSFUNC() has to decide what to tell the SAS processor is the type of the value being passed as the first argument to SUBSTRN().  The string you passed it looked like a numeric expression, so that is what it passed.

322   %let calcsets=CA DE GP LE;
323   %put %eval(&calcsets);
0

[Patrick]

@Tom How would you code %sysfunc(substrn(...)) for a source string to not get evaluated when it contains an operator keyword like EQ? 

I also can't think of a use case where it would make sense for the substrn() function to accept a numerical argument. Any ideas?

[Tom]

I don't know of anyway to avoid %SYSFUNC() unmasking your strings.  There are many examples of issues on this forum where that is the cause of trouble.

As to why SUBSTRN() allows a numeric first argument I have no idea.  Perhaps they were just caught up in the whole idea of that new functionality used in the CAT...() series of functions and decided to include it into the new SUBSTRN() function also.

[Quentin]

Agree, the more times this problem pops up, the more I think it was a mistake for SAS to design these newer functions so that they could take either a numeric or character argument.  Implicit numeric to character conversions are typically not a good thing.  SAS having to look at a value and then guess based on the content of the value whether it is a string or numeric expression just seems inherently fiddly. 

I guess some of the "type guessing" is just inherent to the macro language design decision not to separately define string values and numeric values.  So %eval called implicitly by %if also needs to look at values / expressions and guess whether or not they are numeric.

At least with %eval, we can use quoting functions to protect a string value that looks like a number.  I don't think the quoting functions can help with %sysfunc(), it seems either %sysfunc or the function it is calling unquotes the values during execution before it guesses whether a value is a string or number.