SASPy and the Macro Processor

[mrmcdonald]

Hi Experts, I've been working to modernize our front-end applications using Python (with the PyQt framework) and connecting it to our existing back-end SAS programs. Typically, I've been successful in calling the back-end SAS programs by using %INCLUDE inside the submit() saspy method. Our back-end SAS programs utilize the macro language pretty extensively.

For example, the back-end programs use macro variable calls like &&&PRODUCT._ATTRIBUTE_SPEC_LIMIT which first resolves to &PROD1_ATTRIBUTE_SPEC_LIMIT which then resolves to a value like 95. In SAS, I can run the following code where both macro calls resolve to 95:

%LET PRODUCT = PROD1;
%LET PROD1_ATTRIBUTE_SPEC_LIMIT = 95;

%PUT Triple Ampersand result is &&&PRODUCT._ATTRIBUTE_SPEC_LIMIT;
%PUT Single Ampersand result is &PROD1_ATTRIBUTE_SPEC_LIMIT;

However, in Python using the SASPy module, it appears only 1 "loop" is being run in the macro processor (instead of resolving until there's no &). Thus, &&&PRODUCT._ATTRIBUTE_SPEC_LIMIT is only being resolved once to &PROD1_ATTRIBUTE_SPEC_LIMIT and not 95. The following code could be pasted in Python provided the sas session object named "sas" has already been created:

import saspy

sas.symput("PRODUCT", "PROD1")
sas.symput("PROD1_ATTRIBUTE_SPEC_LIMIT", "95")
sas.submit_return_value = sas.submit(
    """
    %PUT Triple Ampersand result is &&&PRODUCT._ATTRIBUTE_SPEC_LIMIT;
    %PUT Single Ampersand result is &PROD1_ATTRIBUTE_SPEC_LIMIT;
    """,
results = 'LIST',
log = 'LIST'
)
print(sas_submit_return_value['LOG'])

Is there something I may be missing? I would like to avoid touching the numerous back-end SAS programs if at all possible. Thanks for the help!

[ballardw]

The indirect references (i.e. the &&&&&&& type of references) in general can be fun to debug.

Just of giggles have you tried 4 & version?

[mrmcdonald]

Not quite, what do you mean? I'm using versions SAS 9.4M3, Python 3.11, and SASPy 5.4.0. I've used MLOGIC, SYMBOLGEN, and MPRINT. Using SAS "directly", the macros are resolving but not when we use SASPy.

[AhmedAl_Attar]

@mrmcdonald 

What happens if you use

%put %superq(&PRODUCT._ATTRIBUTE_SPEC_LIMIT);

OR 

%put _user_;

[mrmcdonald]

I get an error message when I try a single &.
SYMBOLGEN: Macro variable PRODUCT resolves to PROD1
SYMBOLGEN: Some characters in the above value which were subject to macro quoting have been unquoted for printing.
ERROR: Invalid symbolic variable name PROD1.

[Quentin]

---

@mrmcdonald wrote:
I get an error message when I try a single &.
SYMBOLGEN: Macro variable PRODUCT resolves to PROD1
SYMBOLGEN: Some characters in the above value which were subject to macro quoting have been unquoted for printing.
ERROR: Invalid symbolic variable name PROD1.

---

That log makes no sense to me at all.  It's as if a single ampersand is somehow triggering a rescan. 

[AhmedAl_Attar]

@mrmcdonald 

Here is what I was able to get from my SASPy session

>>> sas=saspy.SASsession(cfgname='ssh')
Pandas module not available. Setting results to HTML
*** bcc-inf-idns01.tco.census.gov can't find 20HQPLSU4464016: Non-existent domain
SAS Connection established. Subprocess id is 8968

>>> sas.symput('PRODUCT','PROD1')
>>> sas.symput('PROD1_ATTRIBUTE_SPEC_LIMIT','95')
>>> ll = sas.submit('%put _user_')
>>> print(ll['LOG'])

31   ods listing close;ods html5 (id=saspy_internal) file=stdout options(bitmap_mode='inline') device=svg style=HTMLBlue; ods
31 ! graphics on / outputfmt=png;
NOTE: Writing HTML5(SASPY_INTERNAL) Body file: STDOUT
32
33   %put _user_
34
35   ods html5 (id=saspy_internal) close;ods listing;
GLOBAL PROD1_ATTRIBUTE_SPEC_LIMIT 95
GLOBAL PRODUCT PROD1

>>> py_val = sas.symget('PROD1'+'_ATTRIBUTE_SPEC_LIMIT')
>>> py_val
95

Note: I'm running Python 3.9.6 (on my laptop) and SAS 9.4 M8 (on my Linux Server)

The %put _user_ is just to ensure the macro variables were correctly defined in SAS

The sas.symget clause to illustrate how to use Python string literal with SAS

I'm aware this does not resolve your macro resolution issue, but you may want to rethink the way to go about it!? Here is a link to samples on how to use symput & symget with SASPy

Using_SYMGET_and_SYMPUT.ipynb  

Hope this helps,

Ahmed

[Quentin]

I don't think you're missing anything, and would suggest contacting tech support.

But I'm curious, if you turn on system option SYMBOLGEN, does that provide any interesting information in the log about attempts to resolve macro variables?  I don't use saspy, but your SAS code shows a clear resolution process, consistent with your explanation of how it should work:

1    options symbolgen ;
2    %LET PRODUCT = PROD1;
3    %LET PROD1_ATTRIBUTE_SPEC_LIMIT = 95;
4
5    %PUT Triple Ampersand result is &&&PRODUCT._ATTRIBUTE_SPEC_LIMIT;
SYMBOLGEN:  && resolves to &.
SYMBOLGEN:  Macro variable PRODUCT resolves to PROD1
SYMBOLGEN:  Macro variable PROD1_ATTRIBUTE_SPEC_LIMIT resolves to 95
Triple Ampersand result is 95
6    %PUT Single Ampersand result is &PROD1_ATTRIBUTE_SPEC_LIMIT;
SYMBOLGEN:  Macro variable PROD1_ATTRIBUTE_SPEC_LIMIT resolves to 95
Single Ampersand result is 95

[mrmcdonald]

Thanks for the reply! My log (using SASPy) displays the following:
OPTIONS SYMBOLGEN;

SYMBOLGEN: && resolves to &.
SYMBOLGEN: Macro variable PRODUCT resolves to PROD1
SYMBOLGEN: Some characters in the above value which were subject to macro quoting have been unquoted for printing.
%PUT Triple Ampersand result is &&&PRODUCT._ATTRIBUTE_SPEC_LIMIT;
Triple Ampersand result is &PROD1_ATTRIBUTE_SPEC_LIMIT

[Quentin]

I'd say that log fits with your thought that && isn't actually triggering a re-scan like it should.  I would send the code with the SYMBOLGEN option and the log to tech support, and see what they have to say.

[Tom]

What does the LOG returned by sas.submit() show you?
Why are you passing 'LIST' as the value for both RESULTS and LOGS option?  Is that is just saying "YES I want those to be returned"? Or is the name you want it to assign to them?  If the latter is it an issue if you use the same name for both?

Another debugging test is to make sure the macro variables really exist.  Try using a data step instead of macro code to display the values.

data _null_;
  product = symget('product');
  limit = symget(cats(product,'_attribute_spec_limit'));
  put product= / limit=;
run;

[sastpw · Accepted solution]

I think that all you need to do to get the triple ampers to resolve as you're expecting is to specify quoting=None on the first symput call. The default is to use NRBQUOTE when defining the macro variable, which is 'usually' the most helpful, but with SAS/Marco there's always any number of weird things you need to deal with. See the doc for symput() here: https://sassoftware.github.io/saspy/api.html#saspy.SASsession.symput

Here's what I see with and without that:

>>> sas.symput("PRODUCT1", "PROD1");print(sas.lastlog())

21
22 %let PRODUCT1=%NRBQUOTE(PROD1);
23
24
>>> sas.symput("PROD1_ATTRIBUTE_SPEC_LIMIT", "96");print(sas.lastlog())

26
27 %let PROD1_ATTRIBUTE_SPEC_LIMIT=%NRBQUOTE(96);
28
29
>>> sas.submitLOG('''
... %PUT Triple Ampersand result is &&&PRODUCT1._ATTRIBUTE_SPEC_LIMIT;
... %PUT Single Ampersand result is &PROD1_ATTRIBUTE_SPEC_LIMIT;
... ''')

31 ods listing close;ods html5 (id=saspy_internal) file=stdout options(bitmap_mode='inline') device=svg style=HTMLBlue; ods
31 ! graphics on / outputfmt=png;
NOTE: Writing HTML5(SASPY_INTERNAL) Body file: STDOUT
32
33
34 %PUT Triple Ampersand result is &&&PRODUCT1._ATTRIBUTE_SPEC_LIMIT;
Triple Ampersand result is &PROD1_ATTRIBUTE_SPEC_LIMIT
35 %PUT Single Ampersand result is &PROD1_ATTRIBUTE_SPEC_LIMIT;
Single Ampersand result is 96
36
37
38 ods html5 (id=saspy_internal) close;ods listing;

>>> sas.symput("PRODUCT2", "PROD2", quoting=None);print(sas.lastlog())

40
41 %let PRODUCT2=PROD2;
42
43
>>> sas.symput("PROD2_ATTRIBUTE_SPEC_LIMIT", "96");print(sas.lastlog())

45
46 %let PROD2_ATTRIBUTE_SPEC_LIMIT=%NRBQUOTE(96);
47
48
>>> sas.submitLOG('''
... %PUT Triple Ampersand result is &&&PRODUCT2._ATTRIBUTE_SPEC_LIMIT;
... %PUT Single Ampersand result is &PROD2_ATTRIBUTE_SPEC_LIMIT;
... ''')

50 ods listing close;ods html5 (id=saspy_internal) file=stdout options(bitmap_mode='inline') device=svg style=HTMLBlue; ods
50 ! graphics on / outputfmt=png;
NOTE: Writing HTML5(SASPY_INTERNAL) Body file: STDOUT
51
52
53 %PUT Triple Ampersand result is &&&PRODUCT2._ATTRIBUTE_SPEC_LIMIT;
Triple Ampersand result is 96
54 %PUT Single Ampersand result is &PROD2_ATTRIBUTE_SPEC_LIMIT;
Single Ampersand result is 96
55
56
57 ods html5 (id=saspy_internal) close;ods listing;

>>>

Hope that helps,

Tom

[Tom]

That would explain it.

It also explains why the SAS log showed that message about having unquoting something.

SYMBOLGEN:  Some characters in the above value which were subject to macro quoting have been unquoted for printing.

You can recreate the issue using SAS code by adding the %NRBQUOTE() when defining PRODUCT.

1    options symbolgen ;
2    %LET PRODUCT = %nrbquote(PROD1);
3    %LET PROD1_ATTRIBUTE_SPEC_LIMIT = 95;
4
5    %PUT Triple Ampersand result is &&&PRODUCT._ATTRIBUTE_SPEC_LIMIT;
SYMBOLGEN:  && resolves to &.
SYMBOLGEN:  Macro variable PRODUCT resolves to PROD1
SYMBOLGEN:  Some characters in the above value which were subject to macro quoting have been unquoted for printing.
Triple Ampersand result is &PROD1_ATTRIBUTE_SPEC_LIMIT

[Quentin]

Ah, the dreaded %NRBQUOTE.  It's so easy to think the NR stands "no-resolve" but in fact it stands for "no-rescan" so this lack of making a second pass to fully resolve the text makes perfect sense now. Thanks!

Personally, I've never managed to find even a contrived use-case where %NRBQUOTE was more helpful than %BQUOTE, so I'm surprised that SASPy developers would decide to choose it as a default rather than %BQUOTE.  If anyone does have an example where %NRBQUOTE provides important functionality that cannot be obtained by %BQUOTE, please share.

Generally, the idea of automatically adding macro quoting feels a bit foreign to SAS.  I can't think of another example where values are quoted without the explicit using of a quoting function.  But that said, the world of macro quoting is a murky place. So perhaps there was good reason to add automatic quoting, especially when passing text between languages (python and SAS).

[sastpw]

Hey sorry for the confusion. I can speak to 'why' the SASPy developer chose this, as that's me. One of the goals of SASPy is for it to be intuitive for Python developers and not require intimate knowledge of SAS, it's syntax and it's idiosyncrasies. That's a goal, and clearly not 100% achievable, since SAS has so many options and syntaxes ... Still, the provided methods try to be Pythonic instead of SAS-centric. With the submit methods, you can code anything the SAS way, so you always have that.

The goal of symput is to take the value of a Python variable/literal, and make that value exist in SAS as a macro variable. The default for this method is then to make the string from Python be that same string in SAS. Without using the quoting functions, no end of various characters will be parsed/interpreted/changed thus not having the actual Python value be what's in SAS. Of course, providing the option to allow SAS coders to specify special SAS'ism cases is available on the method; thus providing the ability to still get whichever different behavior you want from SAS if you understand the macro language and interpreter. But the default allows real python values to be transferred over without SAS changing them. There's more explanation here: Understanding Why Macro Quoting Is Necessary 

I hope that helps provide perspective on why the default wasn't what you were expecting. 

Thanks!

Tom