SasPy: trouble retrieving variable from macro using .symget()

[restin]

Hi every one,

I'm quite new to SAS. I have a .csv file with data that needs to be processed using a pre-made SAS macro (found here).

I'm using SasPy from a Jupyter notebook with SAS OnDemand for Academics (SAS version 9.4).

I manage to run this macro directly in SAS Studio, with the desired results outputted. However, in a Jupyter notebook setup I do not manage to retrieve the calculated values in the macro that are stored in certain variables.

Hereby the code:

Since the macro is quite lengthy, I first read the source .sas file ('metadas.sas') into a string ('macro'):

macro = open('/Users/user/metadas.sas').read()

In order to run the macro (named '%metadas') on the input .csv file, I need to call it using this SAS code (also providing a path for the output .rtf and .log file):

%metadas(
dtfile='/saswork/SAS_work559F000186B8_odaws01-euw1.oda.sas.com/SAS_work7C27000186B8_odaws01-euw1.oda.sas.com/test.csv',
rfile='/saswork/SAS_work559F000186B8_odaws01-euw1.oda.sas.com/SAS_work7C27000186B8_odaws01-euw1.oda.sas.com/test.rtf',
logfile='/saswork/SAS_work559F000186B8_odaws01-euw1.oda.sas.com/SAS_work7C27000186B8_odaws01-euw1.oda.sas.com/test.log');

So I concatenate the macro string and the command string into a single string, which I then submit to the SAS session:

input_csv = '/saswork/SAS_work559F000186B8_odaws01-euw1.oda.sas.com/SAS_work7C27000186B8_odaws01-euw1.oda.sas.com/test.csv'
output_rtf = '/saswork/SAS_work559F000186B8_odaws01-euw1.oda.sas.com/SAS_work7C27000186B8_odaws01-euw1.oda.sas.com/test.rtf'
output_log = '/saswork/SAS_work559F000186B8_odaws01-euw1.oda.sas.com/SAS_work7C27000186B8_odaws01-euw1.oda.sas.com/test.log'

code_subm = ("""{} \n %metadas(dtfile='{}',rfile='{}',logfile='{}');""").format(macro, input_csv, output_rtf, output_log)
sas.submit(code_subm)

One of the calculated variables within the macro that I need is, 'senspred'. However, when I try to access that variable:

res = sas.symget('senspred')
print(res)

Python returns the variable name with ampersand, instead of the value of that variable:

&senspred

I get the same result with all other macro variables that I try to access (including the inputted variables dtfile, rtfile, logfile).

I used these sources as an example:

https://github.com/sassoftware/saspy-examples/blob/main/SAS_contrib/Using_SYMGET_and_SYMPUT.ipynb

https://sassoftware.github.io/saspy/advanced-topics.html#moving-values-between-python-variables-and-...

What am I'm missing?

[ChrisHemedinger]

Hi @restin,

There's a lot going on here but I see some opportunities to simplify, and that might help you get further.

First, if you are using the macro as is, then you can consider just reading it in directly from the website:

filename metamac url "https://methods.cochrane.org/sites/methods.cochrane.org.sdt/files/public/uploads/METADAS_v1.3_txt.txt";
%include metamac;

You could sas.submit that code from your notebook, and then you'd have the macro programs available in your session.

Next: hardcoding the work directory path is probably not the best idea as it will change every time you start a new session. If you don't mind storing this output in your SAS OnDemand account, allocate a new folder there and use it. Here's a simple trick (again, using sas.submit):

options dlcreatedir;
libname results "&userdir./results";
libname results clear;

You'll then have a new folder in your account named "results", which you can always reference in your SAS code as "&userdir./results".

Assuming your CSV file is "local" to your notebook session, you'll need to use sas.upload to get this up to SAS. You can place this in your userdir/results for convenience if you want.

With all of that in place, you'll be able to then use sas.submit to run your macro. Note that if you rely on the &userdir macro you'll need the doublequotes in your SAS code so the SAS macro processor will resolve the macro vars. (Single quotes would prevent this.)

%metadas(dtfile="%userdir/results/test.csv",
         rfile="%userdir/results/test.rtf",
         logfile="%userdir/results/test.log");

.

[restin]

Hi Chris,

Thank you for your suggestions. This is the adapted Python code:

# Include METADAS.sas
sas.submit("""
filename metadas url "https://methods.cochrane.org/sites/methods.cochrane.org.sdt/files/public/uploads/METADAS_v1.3_txt.txt";
%include metadas;
""")

# Create results dir
sas.submit("""
options dlcreatedir;
libname results "&userdir./results";
libname results clear;
""")

# List directories in homefolder:
dl = sas.dirlist(sas.workpath)
for i in dl:
    print(i)

# Upload CSV file:
sas.upload("/Users/user/test.csv", str(sas.workpath + "results/test.csv"))

# Execute metadas macro:
sas.submit("""
%metadas(dtfile="%userdir/results/test.csv",
         rfile="%userdir/results/test.rtf",
         logfile="%userdir/results/test.log");
""")
# retrieve value of variable 'senspred'
res = sas.symget("senspred")
print(res)

Printing the directories in sas.workpath seems to show that the 'results' directory hasn't been created, so the rest of macro doesn't seem to work either:

sastmp-000000008.sas7butl
sasgopt.sas7bcat
sasmac4.sas7bcat
sastmp-000000002.sas7bitm
regstry.sas7bitm
sasmac1.sas7bcat
sas.lck
&senspred

What is going wrong?

I made a slightly more elaborate code, that does seem to be able to create the results folder and starts up the macro (using Python's format function and 'sas.workpath'):

# Import METADAS.sas
sas.submit("""
filename metadas url "https://methods.cochrane.org/sites/methods.cochrane.org.sdt/files/public/uploads/METADAS_v1.3_txt.txt";
%include metadas;
""")

# Create results dir
sas.submit(("""
options dlcreatedir;
libname results "{}/results";
libname results clear;
""").format(str(sas.workpath)))

# List directories in homefolder:
dl = sas.dirlist(sas.workpath)
for i in dl:
    print(i)

# Upload CSV file:
sas.upload("/Users/user/test.csv", str(str(sas.workpath) + "results/test.csv"))

# Execute metadas macro:
output_dict = sas.submit(("""
options dlcreatedir;
%metadas(dtfile="{}results/test.csv",
         rfile="{}results/test.rtf",
         logfile="{}results/test.log");
""").format(sas.workpath, sas.workpath, sas.workpath))

res = sas.symget("senspred")
print(res)

Output:

_metadas_warnings.sas7bdat
_metadas_log.sas7bdat
templat.sas7bitm
sastmp-000000048.sas7bitm
_metadas_a_eigenvals_1.sas7bdat
_metadas_modfail.sas7bdat
_metadas_errors.sas7bdat
_metadas_a_hessian_1.sas7bdat
_metadas_ds1.sas7bdat
_metadas_meta.sas7bdat
test.csv
parms.sas7bcat
results/
sastmp-000000008.sas7butl
sasgopt.sas7bcat
sasmac4.sas7bcat
sastmp-000000002.sas7bitm
regstry.sas7bitm
sasmac1.sas7bcat
sas.lck

101                                                        The SAS System                           zondag 21 november 2021 15:49:00

21874      ods listing close;ods html5 (id=saspy_internal) file=_tomods1 options(bitmap_mode='inline') device=svg style=HTMLBlue;
21874    ! ods graphics on / outputfmt=png;
21875      
21876      
21877      options dlcreatedir;
21878      %metadas(dtfile="/saswork/SAS_workFCF800002BDA_odaws01-euw1.oda.sas.com/SAS_workC59300002BDA_odaws01-euw1.oda.sas.com/tes
21878    ! t.csv",
21879               rfile="/saswork/SAS_workFCF800002BDA_odaws01-euw1.oda.sas.com/SAS_workC59300002BDA_odaws01-euw1.oda.sas.com/resu
21879    ! lts/test.rtf",
21880               logfile="/saswork/SAS_workFCF800002BDA_odaws01-euw1.oda.sas.com/SAS_workC59300002BDA_odaws01-euw1.oda.sas.com/re
21880    ! sults/test.log");
*******************************************************
*                                                  *
*  META-ANALYSIS OF DIAGNOSTIC ACCURACY STUDIES  *
*                       *
*******************************************************
21923      
21924      
21925      
21926      ods html5 (id=saspy_internal) close;ods listing;
21927      

102                                                        The SAS System                           zondag 21 november 2021 15:49:00

21928      
&senspred

Here the macro seems to have create some intermediate files. I'm still not able to print the value of variable 'senspred' however.

Attached the created 'test.log' file. It mentions a syntax error, referring to the path of the 'test.csv' file, but I don't see what's wrong with it.

Any further help much appreciated.

[ChrisHemedinger]

I might have led you down the wrong path with that &userdir macro var -- that's a SAS Studio thing and obviously you're not using that. Instead of that, try this for the path:

/home/&sysusername./results

If that doesn't work, just sub in your actual SAS OnDemand user ID which will be a "uNNNNNNNN" style token:

/home/u12345678/results

[ChrisHemedinger]

Also,  I scanned the code in the macro you reference and I didn't see anything that would assign the senspred macro variable. No call symput or let statements that reference it. It seems to be a variable in some output data, but never stored in a macro variable. If that's the case, then the symget call wouldn't yield anything.

[Kurt_Bremser]

Linux (UNIX) provides two shortcuts to your home directory: the tilde (~) or the environment variable $HOME, so you can write your statement as

libname results "~/results";

or

libname results "$HOME/results";

[restin]

I tried all four options with no success:

sas.submit("""
options dlcreatedir;
libname results "$HOME/results";
libname results clear;
""")

sas.submit("""
options dlcreatedir;
libname results "~/results";
libname results clear;
""")

sas.submit("""
options dlcreatedir;
libname results "/home/&sysusername./results";
libname results clear;
""")

sas.submit("""
options dlcreatedir;
libname results "/home/U59923716/results";
libname results clear;
""")

When I execute this Python code, no results folder shows up:

# List directories in homefolder:
dl = sas.dirlist(sas.workpath)
for i in dl:
    print(i)

Is sas.workpath the same as the home folder in which we created the results folder?

But leaving this aside, as said, I could get the macro partially running by using sas.workpath to create a results folder, instead of the home folder.

Also,  I scanned the code in the macro you reference and I didn't see anything that would assign the senspred macro variable. No call symput or let statements that reference it. It seems to be a variable in some output data, but never stored in a macro variable. If that's the case, then the symget call wouldn't yield anything.

OK, but even if I try to retrieve other macro variables that do get assigned a value, like 'tp', 'cinterval', I get the same results (ampersand added before the variable but not the value of the variable):

res = sas.symget("cinterval")
Python output:
&cinterval

res = sas.symget("tp")
Python output:
&tp

In addition, I get the beneath error in the log file: anyone an idea on how to resolve this? I suppose this is the error because of which the macro doesn't make it to the end:

******************************
CREATE WORD OUTPUT FOR RESULTS
******************************
5450 "^S={outputwidth=100% just=l font_size=3} Data:
5450 ! "/saswork/SAS_work7B650001B98F_odaws02-euw1.oda.sas.com/SAS_workCAA40001B98F_odaws02-euw1.oda.sas.com/test.csv"
_
22
200
ERROR 22-322: Syntax error, expecting one of the following: ;, ANCHOR, AUTHOR, BASE, BODY, BOOKMARK, BOX_SIZING, CHARSET, CLOSE,
COLUMNS, CONTENTS, CSSSTYLE, DATAPANEL, DOM, DPI, ENCODING, FILE, FONTSCALE, GFOOTNOTE, GTITLE, HOST, IMAGE_DPI,
KEEPN, LSTPIPE, NEWFILE, NOCONTENTS, NOGFOOTNOTE, NOGTITLE, NOKEEPN, NOLSTPIPE, NOOUTLINE, NOTRKEEP, OPERATOR,
OUTLINE, PACKAGE, PATH, RECORD_SEPARATOR, SAS, SASDATE, SGE, STARTPAGE, STYLE, TEXT, TITLE, TRANTAB, TRKEEP.

thanks!