SCAPROC: Generating Parallel Code for SAS Viya

2 Likes

Previously, in my last blog: Unearthing Secrets Hidden Within Grid-Enabled Code, I took a look at the SAS Code Analyzer Procedure (SCAPROC). My focus was on SAS 9 grid environments and how SCAPROC helps create SAS/CONNECT-based parallelized SAS code as output from a non-parallel input. We discovered how useful SCAPROC can be on a SAS 9 grid environment and noted some interesting “hidden/internal” commands that SCAPROC’s grid-enabled code utilizes that I hadn’t seen anywhere else (they now have SAS 9.4 & Viya 3.5 documentation! at: SAS Help Center: Concepts PROC SCAPROC).

Toward the end of that blog, I mistakenly wrote that SAS Viya would not utilize SCAPROC. My day-to-day work (and therefore my focus) has been on SAS 9 Grid, and I have some knowledge of CAS in Viya... so I thought CAS on SAS Viya was the mechanism you would utilize to do your parallel processing. I amended that section of the blog after my colleague Bruno Mueller pointed me down the path of really testing in Viya, where it seems we can use multiple SAS/CONNECT sessions to achieve parallelization with SCAPROC (much like what you would see on SAS 9 grid with multiple compute servers).

After my assumptions were shattered, I shared then that I’d like to explore this topic further, and this blog is the continuation of that conversation!

SCAPROC?

For those who don’t know and don’t want to do a quick skim of my last blog (how dare you), I’ll very briefly cover the SCAPROC procedure again here. For those that know what's up and just want to see what happens when I run it feel free to skip this section.

So, SAS Code Analyzer Procedure... You take some SAS code, and wrap it in SCAPROC procedures. You start with a record statement and end with a write statement, it looks something like this:

proc scaproc;
    record ; /* Stats and comments */
run;

/* Your sas code goes here!!!! */

proc scaproc;
    write;
run;

The record file is what gets created and filled with generated comments describing your code. There’s also a “grid” option under the record statement that, when utilized, lets you specify a grid code file as well so that SCAPROC will take your SAS code and comment it while also parallelizing for grid services.

Upon hitting “Run”, SAS will then analyze your code! That analysis informs the creation of the “record” and “grid” code files. If you want more details, check out the Viya platform documentation for SCAPROC here: SAS Help Center: Syntax: PROC SCAPROC PROC SCAPROC Statement

SCAPROC... in SAS Viya this time!

The following behavior was observed on SAS Viya (4.x) using compute sessions with SAS/CONNECT enabled.

I got myself a SAS Viya environment fired up and before I even started, I knew I needed a few things.

A location to write and read my SCAPROC-generated code files to/from
A library that was accessible across all SAS/CONNECT sessions

Let's go in order and tackle bullet point one. Thankfully, the trusty work directory is always there for me (and all users). These generated code files, at least for the purpose of this testing, can be temporary. I used the system function to grab the local WORK location and threw my “record” and “grid” SCAPROC files in there, it looked like this:

%let WORKDIR=%sysfunc(getoption(work));
proc scaproc;
    record "&WORKDIR/comments.sas" /* Stats and comments */
    grid "&WORKDIR/grid.sas" /* grid-enabled code */
run;

/* I'll write some SAS code here give me a moment */

proc scaproc;
    write;
run;

Now I needed to read those created files. Normally you would just write them to a location accessible in your user interface (I was in SAS Studio) and open the files there once they’d been created... but my test environment had some read/write lockdown restrictions. Instead of going into full admin mode to dive into the SAS Viya configs and remove the restrictions, I decided to play the part of a user whose admin was unavailable that day. This meant I needed to read from that work location, and I got to try out the ‘put’ statement for use with files! Check this out:

data _null_;
    infile "&WORKDIR/comments.sas" truncover;
    input;
    put _infile_;
run;

data _null_;
    infile "&WORKDIR/grid.sas" truncover;
    input;
    put _infile_;
run;

Tacking on these data steps at the end of my SCAPROC code would mean the contents of comments.sas and grid.sas would be output to the log.

Perfect! I can read from the log!

You can even do “put _n_ = _infile_;” to get line numbers, mostly to see if the log output is hiding anything from you at the top of the file (it wasn’t).

Now we can add some SAS code for our SAS Code Analyzer Procedure to analyze! I decided on using SASHELP datasets, as the SASHELP library would be accessible to the SAS/CONNECT sessions that grid.sas would be starting up. However, SASHELP is read only; I needed another library to write the outputs to. This other library had to be shared across connect sessions. This is what I was referring to in bullet point two earlier. The library was created with the name “gshared” and placed in an RWX persistent volume so that it was accessible to other Kubnetes pods (and therefore accessible by both our local machine and SAS/CONNECT sessions we’d be starting up in grid.sas).

libname gshared "/data/gridwork";

We need a libname statement to run on each of the SAS/CONNECT sessions we’ll be creating, so within the SCAPROC wrapper functions I included this piece of code that will run on all SAS/CONNECT sessions:

/* SCAPROC GLOBAL BEGIN */;
options dlcreatedir;
libname gshared "/data/gridwork";
/* SCAPROC GLOBAL END */;

(check out the documentation to learn more about this SCAPROC GLOBAL syntax: SAS Help Center: Concepts PROC SCAPROC)

I ran the univariate procedure on the SASHELP.CLASS dataset, and the sort, means, and SQL procedures on the SASHELP.CARS datasets. This is what the final product looked like (wrapped with the SCAPROC statements and with those “file put” data steps at the bottom):

Link to full scaprocgenerate.sas file on GitHub: sas-education/articles/SCAPROC-SAS-Viya/scaprocgenerate.sas at main · sassoftware/sas-education Now we just run this and generate comments.sas and grid.sas!

comments.sas:

Link to full comments.sas file on GitHub: sas-education/articles/SCAPROC-SAS-Viya/comments.sas at main · sassoftware/sas-education

comments.sas has the four procedure steps from scaprocgenerate.sas with surrounding /* JOBSPLIT */ comments to describe what’s going on underneath the hood.

You see things like temporary inputs, library locations, and time markers.

Viewing this output can give you a good idea of how SAS is handling your code.

grid.sas:

Link to full grid.sas file on GitHub: sas-education/articles/SCAPROC-SAS-Viya/grid.sas at main · sassoftware/sas-education

We’re more interested in running grid.sas for this blog, but it’s nice to see the contents of both files.

When we do run grid.sas we encounter an error:

%let rc=%sysfunc(grdsvc_enable(_all_, resource=SASApp));
ERROR: The HTTP request failed.
NOTE: The specified SAS/CONNECT context "SASApp" is invalid.
NOTE: SAS application server not found; the local machine will be used.

It seems SCAPROC makes an assumption that is normally valid on SAS 9 environments, but is not on SAS Viya... we don’t have a ‘SASApp’ SAS application server resource. We can add an extra option, ‘resource’, in our PROC SCAPROC statement to specify our default SAS/CONNECT context (or we could specify any other SAS/CONNECT context we may have created).

proc scaproc;
    record "&WORKDIR/comments.sas" /* Stats and comments */
    grid "&WORKDIR/grid.sas" /* grid-enabled code */
    resource 'default-launcher';
run;

SAS Help Center: SAS/CONNECT Server DATA Step Functions

Once grid.sas is regenerated, it will have “default-launcher” as a resource:

%let rc=%sysfunc(grdsvc_enable(_all_, resource=default-launcher));

Now SAS can enable the grid service via the default SAS Viya connect context! And don’t be confused by the name... there are batch contexts, connect contexts, compute contexts, and then launcher contexts for each of them. Despite being called “default-launcher” this is indeed a connect context.

%scaproc_start_sessions(&SCAPROC_SESSIONS_COUNT);
NOTE: Remote session ID SESS1 will use the grid service _ALL_.
NOTE: Background signon to SESS1 in progress.
NOTE: Remote session ID SESS2 will use the grid service _ALL_.
NOTE: Background signon to SESS2 in progress.
NOTE: Remote session ID SESS3 will use the grid service _ALL_.
NOTE: Background signon to SESS3 in progress.

Three remote sessions are created: SESS1, SESS2, and SESS3. In this example, there are no dependencies between the tasks across sessions, only dependencies within sessions.

SESS1 receives the PROC UNIVARIATE task. Normally PROC UNIVARIATE would write to the ‘results’ tab in SAS Studio if you ran it on the local machine, but this does not seem to happen on SAS Viya from an asynchronous remote submission, so I copied the relevant outputs to a single-row dataset that can be viewed from the local session (since the dataset is placed in our “gshared” library, which is shared by all local and SAS/CONNECT sessions). I tested on SAS 9.4, and outputs to your ‘Results’ and ‘Output Data’ tabs on a remote machine can be viewed locally as I would’ve expected. I’d like to cover this detail in another future blog once I figure out how to make the outputs show up on the local session in the way that I want them to in SAS VIYA... But, for now, I am going to use this workaround.

SESS2 actually receives two “tasks”: a PROC SORT and a PROC MEANS. SCAPROC treats this as a single task because it's two steps in the one remote submission (they are grouped together because the PROC SORT is a dependency of the PROC MEANS). Because there are two PROC steps in this one task, SAS interestingly goes on to refer to the third and final task in grid.sas (the PROC SQL task) as “task 4” in comments. This is consistent with the logic I observed in SAS 9.4, where SCAPROC considers each PROC/DATA step a “task” in some comments while elsewhere referring to a “task” as a single remote submission (perhaps containing multiple steps). This sounds confusing, but it’s just a labeling thing. It can honestly be helpful when stepping through the code, since the names of each task... In this case: “Task 1”, “Task 2”, and “Task 4”, helps you identify how many total PROC/DATA steps you’ve got (and in which task they end up in).

SESS3, as I just mentioned, receives the final PROC SQL task. All three sessions take in data from a SASHELP dataset, produce some output in the form of a new dataset, and write that output into the “gshared” library for us to view locally. After running grid.sas this is what we see:

Conclusion

So yes! You can use SCAPROC in SAS Viya!!! The biggest difference between it and SCAPROC in 9.4 that I can see would be the specifying of a SAS/CONNECT context instead of a SAS Application Server (which ultimately has the same syntax anyway). I haven’t come across usage of SCAPROC on SAS Viya in the wild before, so I’d be curious to know... Have you used SCAPROC in SAS Viya? What sorts of things have you done with it?

Related Links:

Unearthing Secrets Hidden Within Grid-Enabled Code

SAS Help Center: Syntax: PROC SCAPROC PROC SCAPROC Statement

SAS Help Center: Concepts PROC SCAPROC

SAS Help Center: SAS/CONNECT Server DATA Step Functions

Course: Architecture and Administration for SAS® Workload Management on SAS® Viya®