topic Re: SASPy and the Macro Processor in SAS Programming

SASPy and the Macro Processor

mrmcdonald — Mon, 22 Jan 2024 17:57:43 GMT

Hi Experts, I've been working to modernize our front-end applications using Python (with the PyQt framework) and connecting it to our existing back-end SAS programs. Typically, I've been successful in calling the back-end SAS programs by using %INCLUDE inside the submit() saspy method. Our back-end SAS programs utilize the macro language pretty extensively.

For example, the back-end programs use macro variable calls like &&&PRODUCT._ATTRIBUTE_SPEC_LIMIT which first resolves to &PROD1_ATTRIBUTE_SPEC_LIMIT which then resolves to a value like 95. In SAS, I can run the following code where both macro calls resolve to 95:

%LET PRODUCT = PROD1;
%LET PROD1_ATTRIBUTE_SPEC_LIMIT = 95;

%PUT Triple Ampersand result is &&&PRODUCT._ATTRIBUTE_SPEC_LIMIT;
%PUT Single Ampersand result is &PROD1_ATTRIBUTE_SPEC_LIMIT;

However, in Python using the SASPy module, it appears only 1 "loop" is being run in the macro processor (instead of resolving until there's no &). Thus, &&&PRODUCT._ATTRIBUTE_SPEC_LIMIT is only being resolved once to &PROD1_ATTRIBUTE_SPEC_LIMIT and not 95. The following code could be pasted in Python provided the sas session object named "sas" has already been created:

import saspy

sas.symput("PRODUCT", "PROD1")
sas.symput("PROD1_ATTRIBUTE_SPEC_LIMIT", "95")
sas.submit_return_value = sas.submit(
    """
    %PUT Triple Ampersand result is &&&PRODUCT._ATTRIBUTE_SPEC_LIMIT;
    %PUT Single Ampersand result is &PROD1_ATTRIBUTE_SPEC_LIMIT;
    """,
results = 'LIST',
log = 'LIST'
)
print(sas_submit_return_value['LOG'])

Is there something I may be missing? I would like to avoid touching the numerous back-end SAS programs if at all possible. Thanks for the help!

Re: SASPy and the Macro Processor

ballardw — Mon, 22 Jan 2024 18:08:45 GMT

The indirect references (i.e. the &&&&&&& type of references) in general can be fun to debug.

Just of giggles have you tried 4 & version?

Re: SASPy and the Macro Processor

mrmcdonald — Mon, 22 Jan 2024 18:21:08 GMT

Not quite, what do you mean? I'm using versions SAS 9.4M3, Python 3.11, and SASPy 5.4.0. I've used MLOGIC, SYMBOLGEN, and MPRINT. Using SAS "directly", the macros are resolving but not when we use SASPy.

Re: SASPy and the Macro Processor

AhmedAl_Attar — Mon, 22 Jan 2024 18:58:53 GMT

@mrmcdonald

What happens if you use

%put %superq(&PRODUCT._ATTRIBUTE_SPEC_LIMIT);

%put _user_;

Re: SASPy and the Macro Processor

Quentin — Mon, 22 Jan 2024 19:20:19 GMT

I don't think you're missing anything, and would suggest contacting tech support.

But I'm curious, if you turn on system option SYMBOLGEN, does that provide any interesting information in the log about attempts to resolve macro variables? I don't use saspy, but your SAS code shows a clear resolution process, consistent with your explanation of how it should work:

1    options symbolgen ;
2    %LET PRODUCT = PROD1;
3    %LET PROD1_ATTRIBUTE_SPEC_LIMIT = 95;
4
5    %PUT Triple Ampersand result is &&&PRODUCT._ATTRIBUTE_SPEC_LIMIT;
SYMBOLGEN:  && resolves to &.
SYMBOLGEN:  Macro variable PRODUCT resolves to PROD1
SYMBOLGEN:  Macro variable PROD1_ATTRIBUTE_SPEC_LIMIT resolves to 95
Triple Ampersand result is 95
6    %PUT Single Ampersand result is &PROD1_ATTRIBUTE_SPEC_LIMIT;
SYMBOLGEN:  Macro variable PROD1_ATTRIBUTE_SPEC_LIMIT resolves to 95
Single Ampersand result is 95

Re: SASPy and the Macro Processor

mrmcdonald — Mon, 22 Jan 2024 19:34:04 GMT

Thanks for the reply! My log (using SASPy) displays the following:
OPTIONS SYMBOLGEN;

SYMBOLGEN: && resolves to &.
SYMBOLGEN: Macro variable PRODUCT resolves to PROD1
SYMBOLGEN: Some characters in the above value which were subject to macro quoting have been unquoted for printing.
%PUT Triple Ampersand result is &&&PRODUCT._ATTRIBUTE_SPEC_LIMIT;
Triple Ampersand result is &PROD1_ATTRIBUTE_SPEC_LIMIT

Re: SASPy and the Macro Processor

mrmcdonald — Mon, 22 Jan 2024 19:37:12 GMT

I get an error message when I try a single &.
SYMBOLGEN: Macro variable PRODUCT resolves to PROD1
SYMBOLGEN: Some characters in the above value which were subject to macro quoting have been unquoted for printing.
ERROR: Invalid symbolic variable name PROD1.

Re: SASPy and the Macro Processor

Quentin — Mon, 22 Jan 2024 19:38:02 GMT

I'd say that log fits with your thought that && isn't actually triggering a re-scan like it should. I would send the code with the SYMBOLGEN option and the log to tech support, and see what they have to say.

Re: SASPy and the Macro Processor

Quentin — Mon, 22 Jan 2024 19:40:48 GMT

@mrmcdonald wrote:
I get an error message when I try a single &.
SYMBOLGEN: Macro variable PRODUCT resolves to PROD1
SYMBOLGEN: Some characters in the above value which were subject to macro quoting have been unquoted for printing.
ERROR: Invalid symbolic variable name PROD1.

That log makes no sense to me at all. It's as if a single ampersand is somehow triggering a rescan.

Re: SASPy and the Macro Processor

AhmedAl_Attar — Mon, 22 Jan 2024 21:35:54 GMT

@mrmcdonald

Here is what I was able to get from my SASPy session

>>> sas=saspy.SASsession(cfgname='ssh')
Pandas module not available. Setting results to HTML
*** bcc-inf-idns01.tco.census.gov can't find 20HQPLSU4464016: Non-existent domain
SAS Connection established. Subprocess id is 8968

>>> sas.symput('PRODUCT','PROD1')
>>> sas.symput('PROD1_ATTRIBUTE_SPEC_LIMIT','95')
>>> ll = sas.submit('%put _user_')
>>> print(ll['LOG'])

31   ods listing close;ods html5 (id=saspy_internal) file=stdout options(bitmap_mode='inline') device=svg style=HTMLBlue; ods
31 ! graphics on / outputfmt=png;
NOTE: Writing HTML5(SASPY_INTERNAL) Body file: STDOUT
32
33   %put _user_
34
35   ods html5 (id=saspy_internal) close;ods listing;
GLOBAL PROD1_ATTRIBUTE_SPEC_LIMIT 95
GLOBAL PRODUCT PROD1

>>> py_val = sas.symget('PROD1'+'_ATTRIBUTE_SPEC_LIMIT')
>>> py_val
95

Note: I'm running Python 3.9.6 (on my laptop) and SAS 9.4 M8 (on my Linux Server)

The %put _user_ is just to ensure the macro variables were correctly defined in SAS

The sas.symget clause to illustrate how to use Python string literal with SAS

I'm aware this does not resolve your macro resolution issue, but you may want to rethink the way to go about it!? Here is a link to samples on how to use symput & symget with SASPy

Using_SYMGET_and_SYMPUT.ipynb

Hope this helps,

Ahmed

Re: SASPy and the Macro Processor

Tom — Mon, 22 Jan 2024 21:54:34 GMT

What does the LOG returned by sas.submit() show you?
Why are you passing 'LIST' as the value for both RESULTS and LOGS option? Is that is just saying "YES I want those to be returned"? Or is the name you want it to assign to them? If the latter is it an issue if you use the same name for both?

Another debugging test is to make sure the macro variables really exist. Try using a data step instead of macro code to display the values.

data _null_;
  product = symget('product');
  limit = symget(cats(product,'_attribute_spec_limit'));
  put product= / limit=;
run;

Re: SASPy and the Macro Processor

sastpw — Fri, 02 Feb 2024 19:15:14 GMT

I think that all you need to do to get the triple ampers to resolve as you're expecting is to specify quoting=None on the first symput call. The default is to use NRBQUOTE when defining the macro variable, which is 'usually' the most helpful, but with SAS/Marco there's always any number of weird things you need to deal with. See the doc for symput() here: https://sassoftware.github.io/saspy/api.html#saspy.SASsession.symput

Here's what I see with and without that:

>>> sas.symput("PRODUCT1", "PROD1");print(sas.lastlog())

21
22 %let PRODUCT1=%NRBQUOTE(PROD1);
23
24
>>> sas.symput("PROD1_ATTRIBUTE_SPEC_LIMIT", "96");print(sas.lastlog())

26
27 %let PROD1_ATTRIBUTE_SPEC_LIMIT=%NRBQUOTE(96);
28
29
>>> sas.submitLOG('''
... %PUT Triple Ampersand result is &&&PRODUCT1._ATTRIBUTE_SPEC_LIMIT;
... %PUT Single Ampersand result is &PROD1_ATTRIBUTE_SPEC_LIMIT;
... ''')

31 ods listing close;ods html5 (id=saspy_internal) file=stdout options(bitmap_mode='inline') device=svg style=HTMLBlue; ods
31 ! graphics on / outputfmt=png;
NOTE: Writing HTML5(SASPY_INTERNAL) Body file: STDOUT
32
33
34 %PUT Triple Ampersand result is &&&PRODUCT1._ATTRIBUTE_SPEC_LIMIT;
Triple Ampersand result is &PROD1_ATTRIBUTE_SPEC_LIMIT
35 %PUT Single Ampersand result is &PROD1_ATTRIBUTE_SPEC_LIMIT;
Single Ampersand result is 96
36
37
38 ods html5 (id=saspy_internal) close;ods listing;

>>> sas.symput("PRODUCT2", "PROD2", quoting=None);print(sas.lastlog())

40
41 %let PRODUCT2=PROD2;
42
43
>>> sas.symput("PROD2_ATTRIBUTE_SPEC_LIMIT", "96");print(sas.lastlog())

45
46 %let PROD2_ATTRIBUTE_SPEC_LIMIT=%NRBQUOTE(96);
47
48
>>> sas.submitLOG('''
... %PUT Triple Ampersand result is &&&PRODUCT2._ATTRIBUTE_SPEC_LIMIT;
... %PUT Single Ampersand result is &PROD2_ATTRIBUTE_SPEC_LIMIT;
... ''')

50 ods listing close;ods html5 (id=saspy_internal) file=stdout options(bitmap_mode='inline') device=svg style=HTMLBlue; ods
50 ! graphics on / outputfmt=png;
NOTE: Writing HTML5(SASPY_INTERNAL) Body file: STDOUT
51
52
53 %PUT Triple Ampersand result is &&&PRODUCT2._ATTRIBUTE_SPEC_LIMIT;
Triple Ampersand result is 96
54 %PUT Single Ampersand result is &PROD2_ATTRIBUTE_SPEC_LIMIT;
Single Ampersand result is 96
55
56
57 ods html5 (id=saspy_internal) close;ods listing;

>>>

Hope that helps,

Tom

Re: SASPy and the Macro Processor

Tom — Fri, 02 Feb 2024 21:35:00 GMT

That would explain it.

It also explains why the SAS log showed that message about having unquoting something.

SYMBOLGEN:  Some characters in the above value which were subject to macro quoting have been unquoted for printing.

You can recreate the issue using SAS code by adding the %NRBQUOTE() when defining PRODUCT.

1    options symbolgen ;
2    %LET PRODUCT = %nrbquote(PROD1);
3    %LET PROD1_ATTRIBUTE_SPEC_LIMIT = 95;
4
5    %PUT Triple Ampersand result is &&&PRODUCT._ATTRIBUTE_SPEC_LIMIT;
SYMBOLGEN:  && resolves to &.
SYMBOLGEN:  Macro variable PRODUCT resolves to PROD1
SYMBOLGEN:  Some characters in the above value which were subject to macro quoting have been unquoted for printing.
Triple Ampersand result is &PROD1_ATTRIBUTE_SPEC_LIMIT

Re: SASPy and the Macro Processor

Quentin — Sat, 03 Feb 2024 13:59:54 GMT

Ah, the dreaded %NRBQUOTE. It's so easy to think the NR stands "no-resolve" but in fact it stands for "no-rescan" so this lack of making a second pass to fully resolve the text makes perfect sense now. Thanks!

Personally, I've never managed to find even a contrived use-case where %NRBQUOTE was more helpful than %BQUOTE, so I'm surprised that SASPy developers would decide to choose it as a default rather than %BQUOTE. If anyone does have an example where %NRBQUOTE provides important functionality that cannot be obtained by %BQUOTE, please share.

Generally, the idea of automatically adding macro quoting feels a bit foreign to SAS. I can't think of another example where values are quoted without the explicit using of a quoting function. But that said, the world of macro quoting is a murky place. So perhaps there was good reason to add automatic quoting, especially when passing text between languages (python and SAS).

Re: SASPy and the Macro Processor

sastpw — Mon, 05 Feb 2024 15:31:47 GMT

Hey sorry for the confusion. I can speak to 'why' the SASPy developer chose this, as that's me. One of the goals of SASPy is for it to be intuitive for Python developers and not require intimate knowledge of SAS, it's syntax and it's idiosyncrasies. That's a goal, and clearly not 100% achievable, since SAS has so many options and syntaxes ... Still, the provided methods try to be Pythonic instead of SAS-centric. With the submit methods, you can code anything the SAS way, so you always have that.

The goal of symput is to take the value of a Python variable/literal, and make that value exist in SAS as a macro variable. The default for this method is then to make the string from Python be that same string in SAS. Without using the quoting functions, no end of various characters will be parsed/interpreted/changed thus not having the actual Python value be what's in SAS. Of course, providing the option to allow SAS coders to specify special SAS'ism cases is available on the method; thus providing the ability to still get whichever different behavior you want from SAS if you understand the macro language and interpreter. But the default allows real python values to be transferred over without SAS changing them. There's more explanation here: Understanding Why Macro Quoting Is Necessary

I hope that helps provide perspective on why the default wasn't what you were expecting.

Thanks!

Tom

Re: SASPy and the Macro Processor

Quentin — Mon, 05 Feb 2024 15:43:01 GMT

Thanks @sastpw that makes complete sense that automatic macro quoting is a useful feature for Pythonic folks, and that's the goal of SASpy. I wasn't thinking from a Pythonic perspective.

I would still vote for %BQUOTE by default instead of %NRBQUOTE, because I don't think %NRBQUOTE adds any value vs %BQUOTE. And in this case %NRBQUOTE is causing a problem. But that's just quibbling. The choice to have automatic default quoting makes sense, and you've added (and documented) features to allow folks to who understand macro quoting to implement whatever they need.

Re: SASPy and the Macro Processor

sastpw — Mon, 05 Feb 2024 15:53:18 GMT

That's for your understanding @Quentin . From the SAS doc for %BQUOTE, it states: In addition, %NRBQUOTE masks: & %.

So, again, it's just to allow Python strings to be used in SAS as they are, not interpreted/converted as/to something different - by default.

SAS is tough in that there are multiple ways to get the same behavior from different things and to get different behavior from the same thing!

Re: SASPy and the Macro Processor

Tom — Mon, 05 Feb 2024 15:53:24 GMT

It might be useful to add options to the symput python function to add actual quotes to the values. Then the generated macro variables could be used directly in places where SAS syntax needs a quoted value. And if you used single quotes then macro quoting would not be needed.

Say you had the value of O'Malley mentioned in the macro documentation. It would be nice to be able to directly create a macro variable with the value "O'Malley" or 'O''Malley' so that you could use it in code like:

where charvar = &mvar

Re: SASPy and the Macro Processor

Quentin — Mon, 05 Feb 2024 16:39:33 GMT

@sastpw wrote:

That's for your understanding @Quentin . From the SAS doc for %BQUOTE, it states: In addition, %NRBQUOTE masks: & %.

So, again, it's just to allow Python strings to be used in SAS as they are, not interpreted/converted as/to something different - by default.

SAS is tough in that there are multiple ways to get the same behavior from different things and to get different behavior from the same thing!

%NRBQUOTE doesn't actually mask & or % which look to SAS like macro triggers, sadly. I've read the docs on %NRBQUOTE too many times to remember. The current docs (https://go.documentation.sas.com/doc/en/pgmsascdc/9.4_3.4/mcrolref/p0frhtfqvguv78n1owh5wfeopdrj.htm )just say:

The %NRBQUOTE function is useful when you want a value to be resolved when first encountered, if possible, but you do not want any ampersands or percent signs in the result to be interpreted as operators by an %EVAL function.

I think relating to &, it means that %NRBQUOTE will prevent & from meaning AND. So if you have an & that is not a macro trigger (because it's not followed by an alpha character) and you don't want it to be interpreted by %EVAL as meaning AND, then it does that. e.g.:

%macro nrb(meal=);
  %if %nrbquote(&meal)= %str() %then %put meal=&meal..  Eat Something!;
  %else %put meal=&meal.. Stop eating!;
%mend;
%nrb(meal=Franks & Beans)

But if you pass %nrb(meal=Franks&Beans), you'll get the unresolved macro variable warnings for &Beans, because %nrbquote doesn't stop the macro resolution attempt.

I don't understand what the docs mean about percent signs, because I don't think a percent sign is ever an operator to %EVAL.

So to me, the only use-case I've ever found for %NRBQUOTE was when a value has a solitary &. Which I guess is a use-case, just very limited. But I can accept that for SASPY, that could be one less call to tech support, so could be the right decision. : )

Re: SASPy and the Macro Processor

sastpw — Mon, 05 Feb 2024 17:05:58 GMT

I'll say right now that I'm sure you know more about SAS Macro than I do. I've never needed to use it enough to understand all of it's obtuse functionality. As for the doc, yes, it has bit's of info scattered over different pages and sometimes it's misleading or contradicts itself in different places. The doc I was looking at about it saying & and % is just a different page. I don't know if it's accurate or not. I can't say I tested every possibility of macro strings when implementing symput.

But, given that you can choose any or none of the various macro functions when using symput, or code any way you want with submit, and the fact that I don't make breaking changes to SASPy if I can possibly help it, I'm not inclined to change the default behavior for this and take the chance of breaking someone's code that's already working as they want and expect.

Again, I appreciate your understanding on this!

Thanks!

Tom