DOSUBL question in SAS Viya

[PaigeMiller]

I have a macro that calls DOSUBL (actually I got it off the internet, and maybe I modified it a little, I can't remember). The docs say:

DOSUBL Function

Enables the immediate execution of SAS code after a text string is passed.

Category:	Macro
Restriction:	This function is not supported in a DATA step that runs in CAS.

Not supported in a DATA step that runs in CAS. What about running it in %SYSFUNC(DOSUBL( ))? That's not running it in a DATA step. Will it work? I don't have SAS Viya so I can't test it, but this macro might be needed in SAS Viya in the near future.

Here is the macro:

%macro
   expand_varlist /*Returns an expanded variable list and optionally creates an indexed data set of variable names*/
      (
         data  = _LAST_,            /*[R]Input data*/
         var   = _ALL_,             /*[R]Variable List expanded*/
         where = 1,                 /*[R]Where clause to subset OUT=, useful for selecting by a name suffix e.g. where=_name_ like '%_Status'*/
         expr  = nliteral(&name),   /*[R]An expression that can be used to modify the names in the expanded list*/
         keep  = ,                  /*[O]Keep data set option for DATA=*/
         drop  = ,                  /*[O]Drop data set option for DATA=*/
         out   = ,                  /*[O]Output data indexed by _NAME_ and _INDEX_*/
         name  = _NAME_,            /*[R]Name of the variable name variable in the output data set*/
         label = _LABEL_,           /*[R]Name of the variable name label variable in the output data set*/
         index = _INDEX_,           /*[R]Name of the variable index variable in the output data set*/
         dlm   = ' '                /*[R]List delimiter*/
      );
   %local m i;
   %global syscc1;
   %let i=&sysindex;
   %let m=&sysmacroname._&i;
   %do %while(%symexist(&m));
      %let i = %eval(&i + 1);
      %let m=&sysmacroname._&i;
      %end;
   %put NOTE: &=m is a unique symbol name;
   %local rc &m code1 code2 code3 code4;
   %if %superq(out) ne %then %let code3 = %str(data &out(index=(&index &name)); set &out; &index+1; run;);
   %else %do;
      %let out=%str(work._deleteme_);
      %let code3 = %str(proc delete data=work._deleteme_; run;);
      %end;
   %let code1 = 
        %nrstr(options notes=0; proc transpose name=&name label=&label data=&data(obs=0 keep=&keep drop=&drop) out=&out(where=(&where)); 
            var &var; run; %let syscc1=&syscc; );
   %let code2 = %str(proc sql noprint; select &expr into :&m separated by &dlm from &out; quit;);
   %let code4 = %str(options notes=1;);
   %let rc=%sysfunc(dosubl(&code1 &code2 &code3 &code4));
&&&m.
   %mend expand_varlist;

Call to the macro

%let class_x=%expand_varlist(data=&train,var=&class_x); 

[ChrisHemedinger]

@PaigeMiller,

The macro works in SAS Viya (just tested it). The doc indicating "won't run in DATA step in CAS" means that when using in-memory data (CAS libs) and using the features of DATA step that work with that (faster, distributed processing)...DOSUBL is not possible.  But your existing SAS programs and code patterns all work in SAS Viya even when you're not taking advantage of CAS. 

[PaigeMiller]

Thanks @ChrisHemedinger 

So I would imagine that CALL EXECUTE is the same and will work, as the documentation makes the same statement.

Maybe I don't understand when SAS Viya uses CAS, and when it doesn't use CAS. Is there some documentation I could read, or two or three sentences worth of explanation you could write?

[ChrisHemedinger]

Here's a bit I cribbed from the doc:

In the SAS Viya environment, the DATA step runs either locally in a SAS client session or in CAS. When programming with the DATA step in SAS Viya, it is important to understand where the computation is taking place because performance can become an issue with very large data sets. Also, not all DATA step language elements are supported in both environments, so you need to know where the DATA step is processing to know what features are available.

In SAS Viya, the DATA step runs in one of two environments:

• In a SAS client session, where it runs in a single thread using the SAS V9 engine. 
• In a CAS server session, where it runs in a single thread or in multiple threads using the CAS engine

Read more here.

There are several functions that cannot work in CAS, but you can imagine that DOSUBL and CALL EXECUTE -- which rely on (and exploit) how SAS statements run in a certain order -- would not work the same way in a multithreaded mode.

[PaigeMiller]

Okay, goodie gum-drops. Follow-up question:

The doc indicating "won't run in DATA step in CAS" means that when using in-memory data (CAS libs) and using the features of DATA step that work with that (faster, distributed processing)

So with humongous data sets, you want to use CAS and benefit from the faster, distributed processing. So in SAS Viya, the DOSUBL example isn't the problem, its really not executing anything time consuming; but the CALL EXECUTE would need to run with great speed in a data step ... or is it that the CALL EXECUTE commands can be executed in a DATA step run without CAS, and then the code generated from CALL EXECUTE would need to run in CAS (because that code will have to execute on the humonguous data sets) and is that possible to switch from running in single thread mode to multithread mode between PROCs or between DATA steps?

[ChrisHemedinger · Accepted solution]

To use CAS and DATA step, the input data and output must be in CAS libs. The higher performance in CAS benefits from partitioned data, which is an intentional data prep step (the partition CAS action). See this blog post for an example.

I think that many of the use cases for CALL EXECUTE can be addressed differently in CAS. Would CALL EXECUTE still make sense in some cases? Maybe, but it's not something that's been ported to the CAS environment.

Your SAS v9 code can work in SAS Viya without modifications, and then you can adapt it to take advantage of CAS where it makes sense and as you have time. CAS-enabled code can look quite different than your v9 code, but the performance benefits can really shine.

This blog post from @Rick_SAS is also a very handy reference: A list of SAS DATA step functions that do not run in CAS

[PaigeMiller]

Ok, I have just clicked on your very helpful link, and it looks like I can do something like this (pseudo-code for now)

/* This data _null_ runs in SAS Client mode, allowing CALL EXECUTE */
data _null_;
      set small_data_set;
      ... commands using call execute ...
run;

/* Then we switch to CAS mode and run the commands created by CALL EXECUTE  on a very large data set */
/* but which do not contain CALL EXECUTE */

[ChrisNZ]

Hi @ChrisHemedinger 

On the page you linked to, this text is confusing: "the macro code itself executes in the SAS Studio client session" 

The SAS Studio client does not run any code.

One major advantage of SAS storage vs databases is the ability to keep sorted tables. Do you know if running a data step with a BY statement in CAS keeps the sorted order, i.e. if the threads are synchronised so that data coming back is kept/inserted in the sorted order?

[Rick_SAS]

> On the page you linked to, this text is confusing: "the macro code itself executes in the SAS Studio client session" 

> The SAS Studio client does not run any code.

Right. Delete the word "studio." A better phrase might be "the macro preprocessor runs in the SAS session," or "on the SAS client."

[ChrisNZ]

I reported it to the SAS documentation team.

Even this is unclear:

> Right. Delete the word "studio." A better phrase might be "the macro preprocessor runs in the SAS session," or "on the SAS client."

The client here is SAS Studio. SAS is the server.

There is no SAS client, even if that's the terminology I see used over and over in the CAS documentation.

SAS might be the client of the CAS server (is it?), but that's a different discussion than our focus here.

[Rick_SAS]

In SAS Viya, you can submit CAS actions by using different computer languages such as Python, Lua, R, or SAS. When submitting actions, each of these computational environments is called a client for CAS, which is the server that runs the actions. The term "SAS client" is used in the doc to distinguish between submitting actions from SAS or a different language.

You can use the SAS macro preprocessor if and ONLY if you use the SAS client. Python, R, etc cannot resolve SAS macro code, although they can call the same actions that the SAS code can call.

In the doc, the editors/IDEs are not called clients because it is the language that is important, not the IDE. For example, it doesn't matter if you develop and run your Python code in PyDev, PyCharm, Spyder, or a Jupyter notebook. The important thing is that you are using Python to call CAS actions. 

[ChrisNZ]

This makes perfect sense, thank you Rick.

I'm sure there will some confusion in many cases to come that the client does not refer to the dumb GUI anymore.