Unearthing Secrets Hidden Within Grid-Enabled Code

1 Like

What if I told you there were SAS statements out there untouched by humankind? What if I told you our machines were building their own code right under our noses?

Does that scare you?

It shouldn’t, I’ve sensationalized this scenario quite heavily... You’re hooked now though! Might as well read to the end, we can both learn something new.

This short post is primarily about the SAS Code Analyzer Procedure, or SCAPROC, and the auto-generated grid-enabled code it produces. I was interested in what’s happening inside that machine-produced code. I figured if robots take over in the next couple of years, we should at least be able to speak their language.

SAS.. Code.. Analyzer.. Procedure?

PROC SCAPROC (that’s S-C-A-proc) is about as "SAS" as you can get: it’s a SAS procedure that wraps around SAS code to analyze SAS code to then produce more SAS code.

It’s easy to use since SAS is doing all the work! I’ll give an example:

Let's say you have some beautiful SAS code...

Code:

libname EGout "/my/library/location"; 

 

proc sort data=ORSTAR.ORSTARBASE out=EGout.orcat; 

   by Product_Category; 

run; 

  

proc sort data=ORSTAR.ORSTARBASE out=EGout.ORGroup; 

   by Product_Group; 

run; 

  

proc sort data=ORSTAR.ORSTARBASE out=EGout.ORLine; 

   by Product_Line; 

run; 

  

PROC MEANS DATA=EGout.ORCat 

	NOPRINT CHARTYPE NOLABELS NWAY SUM NONOBS; 

	VAR Total_Retail_Price; 

	CLASS Product_Category / ORDER=UNFORMATTED ASCENDING; 

    OUTPUT 	OUT=EGout.CATEGORYSTATS(LABEL="Summary by Product Category") 

	SUM()=	/ AUTONAME AUTOLABEL  WAYS INHERIT; 

RUN; 

  

PROC MEANS DATA=EGout.ORGroup 

	NOPRINT CHARTYPE NOLABELS NWAY SUM NONOBS; 

	VAR Total_Retail_Price; 

	CLASS Product_Group / ORDER=UNFORMATTED ASCENDING; 

    OUTPUT 	OUT=EGout.GROUPSTATS(LABEL="Summary by Product Group") 

	SUM()=	/ AUTONAME AUTOLABEL  WAYS INHERIT; 

RUN; 

  

PROC MEANS DATA=EGout.ORLine 

	NOPRINT CHARTYPE NOLABELS NWAY SUM NONOBS; 

	VAR Total_Retail_Price; 

	CLASS Product_Line / ORDER=UNFORMATTED ASCENDING; 

    OUTPUT 	OUT=EGout.LINESTATS(LABEL="Summary by Product Line") 

	SUM()=	/ AUTONAME AUTOLABEL  WAYS INHERIT; 

RUN;

This code is simple! We sort some tables, then output some information from each table. As a user in this example let's say we crafted this code a long time ago, and we have just made the switch from using base SAS to having a SAS 9.4 Grid Manager environment with multiple host machines. These host machines, in theory, should be able to parallelize the workload as we’re giving SAS three independent tasks. So how can we make that parallelization happens? Or more specifically, how can we utilize this new multi-server environment that we’ve invested in?

You could rewrite all your code by hand to have it utilize multiple host machines, or you could have SCAPROC take care of that mess for you! Wrapping our existing beautiful SAS code in some SCAPROC procedures (and specifying a shared location for our library that can be accessed by all machines on the grid) allows us to generate a grid-enabled version of our old code:

Code:

proc scaproc;

record '~/comments.sas'

grid '~/grid.sas'

resource 'SASApp';
run;

libname EGout "/opt/sas/gridwork";

(all of our old code goes here!)

proc scaproc;

write;

run;

Running this now will submit our code to the SAS Code Analyzer, which will output “comments.sas” and “grid.sas” (as requested in our SCAPROC statement). The “comments.sas” file is a record of comments from the analyzer, which look like this:

Select any image to see a larger version.
Mobile users: To view the images, select the "Full" version at the bottom of the page.

These JOBSPLIT comments are interesting, possibly they could be the subject of a future post, but today we’re concerned primarily with the grid-enabled code present in our newly generated “grid.sas” file.

Here’s the top of that file:

We get some information about the tasks contained withing the job. The SAS Code Analyzer correctly identifies and runs three tasks in parallel due to a lack of dependency between tasks. We could modify the number of SAS/CONNECT sessions, adding more wouldn’t do anything for this job, but we may want to change to two or even just one if our job was very intensive or if the grid environment was already strained. In this case, the job is not intensive, so we can leave it at the default “3” for the most time-efficient parallel processing.

All that goes on below the comment “Please don’t edit anything below this line.” is what I’d like to talk about today. Let's take the plunge!

Going Below The Line

Below that line, we have some “hidden commands” found in SCAPROC procedures that you will see in your generated code, but would not use when manually coding in SAS.

These commands are:

startup
getsession
shutdown
taskwait

startup:

The SCAPROC startup command takes a number of “tasks” and a number of “sessions”.

In this screenshot we can see it correctly identifies that there are six tasks in our original code (three PROC SORT and three PROC MEANS) and then a macro containing the number of SAS/CONNECT sessions, in this case three, is passed to the command. This command registers compute sessions and task IDs so that tasks can be routed to sessions appropriately. It does not start the SAS/CONNECT sessions themselves, that is handled by a macro above the startup command:

getsession:

The getsession command gets the name of a given session, which is later passed in our remote submit (rsubmit) statement.

In this screenshot we grab information for “session 2” and use that information to remote submit task two and task three to that session (SCAPROC groups the two because the sort task’s output feeds into the means task’s input). It’s important to note that “getsession 2” does not mean get TWO sessions, it means get the SECOND session from a list of X available sessions and store it as “sess”.

shutdown:

At the end of the generated code file, a macro is called to wait for the completion of and sign off of all sessions that were previously started. Then, the SCAPROC command “shutdown” is issued:

This command, shutdown, does not stop SAS sessions or handle sign offs (all of that is handled in the scagrid_waitfors “do” loop). The shutdown command is instead a shutdown of SCAGRID functions, as noted in the comment above the invocation SCAPROC statement. This results in a clean termination of SCAPROC’s session tracking which was launched by the “startup” command.

taskwait:

In our generated code in this example, there is currently no need for a taskwait command because tasks ran in different sessions do not depend upon each other... but what would happen if we merged all the tables at the end? Then we would need to make sure each table was sorted and the PROC MEANS was ran before merging, otherwise the merge statement would produce unpredictable and unsatisfactory results.

That’s where the taskwait command comes in:

The SAS Code Analyzer is good at identifying which tasks have dependencies and will comment out which dependencies it detects, as you can see from the top comment in the screenshot. Task dependency is sorted sequentially by number, so this next session and the task within that session only need to worry about task two being finished (task two already waits for task one, so there’s no sense in waiting for it again!). That’s what the taskwait command accomplishes, it allows task-to-task synchronization across sessions so that data access and modification is performed in the order we expect.

Why Do We Care About These SCAPROC Commands?

Apart from the benefit of simply knowing what's going on in the code that you’re running, the knowledge of SCAPROC command in generated grid-enabled code may be particularly valuable for users that are looking to adapt their grid code for use in SAS Viya.

Understanding how SAS code on your SAS 9.4 Grid environment decides on task counts, dependencies, and resolving programs with multiple steps can be highly beneficial when making the move to SAS Viya because knowing how your code is parallelized lets you easily reproduce it (and can even help you innovate on existing code, perhaps by making your tasks more efficiently utilize parallel processing capabilities).

My first thought for SAS Viya parallelization was to use CAS... However, not all SAS workloads are appropriate to be executed in CAS. But that's okay, because I've learned it is possible to sign on to grid and start SAS/CONNECT on multiple sessions within SAS Viya. Remember SAS/CONNECT sessions are very similar in their capabilities to SAS Compute Servers/SAS Programming Runtime Environments. This enables parallel execution of SAS data step code in SAS Viya. So, this knowledge does transfer somewhat directly from SAS 9 since leveraging CAS is not required. ~~Be on the lookout for a future blog about this and more from me~~! --> The blog 😏: SCAPROC: Generating Parallel Code for SAS Viya

You can play with and learn about SAS’s SCAPROC procedure for use on grid in our upcoming SAS 9.4M9 Grid administration workshop! Details coming soon 😉 Reach out if you want to know more!

Related Links:

313-2010: Thoroughly Modern SAS®: The SAS® Code Analyzer Helps Bring Programs Up to Date

PROC SCAPROC: Overview: SCAPROC Procedure

Effective Usage of SAS® Enterprise Guide® in a SAS® 9.4 Grid Manager Environment (by Edoardo Riva)

Find more articles from SAS Global Enablement and Learning here.