Parallel Execution in SAS Studio Flows on Compute

[PhilGUK]

In EG, I discovered the Project Properties: Code Submission = Allow parallel execution on the same server check box.  I used that to develop a project flow that had 3 or 4 code nodes in series, it then branched to run 6 things in parallel, before returning to run in series.  Each of the parallel parts took about 50 mins, so running in parallel saved about 5x50 = 250 min = 4h+.

At the time, I thought I found something online that showed the same kind of thing was available with a switch in SAS Studio flows. This blog from 2016 shows something: SAS Studio Parallel Process Flows - SAS Users However, that has .cpf for flows, where we now have .flw for flows and the properties are not the same.

I have been asked if flows in Studio can be set up to run the same as I had in EG.  I can't find anything that explains if this is possible/how to do it.

Is running different parts of a flow in parallel possible in Studio, as it was in EG?

What I ended up with in the EG Flow was very much like a flow in SAS Management Console.  Separate jobs doing their own thing in a separate WORK session, with SAS MC handling the dependencies.  Where the "jobs" shared data/macro variables, I had to take steps to make sure they had the same environment, macro vars and a "shared" work lib (as a perm library) where they could share datasets that previously were in WORK.  That flow was complex due to the amount of shared items, but the concept was easy enough to understand and apply.

Older versions of Studio appear to have this option, does current version have it hidden away?  We are SAS Studio Version: Stable 2025.09

Many thanks

Phil

[SASKiwi · Accepted solution]

This is off-topic, but if you implemented code-based parallel execution using SAS/CONNECT and the SIGNON, RSUBMIT, ENDRSUBMIT and SIGNOFF statements, this will work equally as well and without any modification in Enterprise Guide, SAS Studio or run/scheduled as a batch job via SAS Management Console. 

Of course if you don't have SAS/CONNECT this won't be an option but I thought I would just point it out. We use this approach extensively and it does work really well.  

[PhilGUK]

Thanks for that reminder.  We do have SAS/Connect, we still have a huge number of user who continue to use BASE, even after we went belatedly to Unix in July 2015 and are now deep in the migration to Viya.

I had tried that approach a couple of years ago in EG, but the performance wasn't doing it for me.  But that may have been due to testing the idea, rather than having a real problem.  Our system was (and still is) very slow to get past "preparing server", typically 2-3 mins and at some times of day 10 mins+, so I may have seen that as a problem, where in the real case I had, the savings of 5 x 50+mins (5x 100mins if run during daytime hours) outweighed any delay on connecting the servers.

I'll pass that approach on and see it we can make it work.

Thank you so much for your fast reply.

I worked with a SAS star from New Zealand when I first started using SAS in a UK financial firm. He was my SAS hero.  I'm wondering if @SASKiwi is you SM?

[SASKiwi]

@PhilGUK - Yes, I have worked in the UK too but for SAS out at Marlow, many years ago. Definitely a career highlight!

Regarding our use cases for SAS/CONNECT, they are typically resource intensive especially high CPU, so the session start up time becomes immaterial. Parallelising one of our jobs reduced run time from 5 or 6 hours down to 2.5 hours so more than halved. This is purely SAS 9.4, Viya not being an option for us as yet.

[LinusH]

Studio on Viya 4 is bot the exact same product as in 9.4/3.5, even if they loko similar.

And ASFAIK there is no such functionality OOTB for this.

You current options would be:

• Stay in EG, it has support for Viya 4
• Program you own version of this, ising MP CONNECT
• Transfer you SMC logic to SAS Job Flows

The DI Studio transformation Fork which is similar to what you describe has not yet been implemented in SAS Studio.

[AhmedAl_Attar]

Hi @PhilGUK 

Considering what you have mentioned "We are SAS Studio Version: Stable 2025.09", it sounds like you are running/using SAS Viya!!

While Lift (v9 code) and Shift (run on Viya compute) is possible, it does not always give you all the performance enhancements that comes with Viya!!

If you have the option/ability to convert your existing/current SAS V9 code to utilize CAS and CASL, you may find yourself not needing explicit Parallel code execution, because CAS and CASL does that implicitly behind the scenes for you and cuts down the 50 min processing to much lower.    

Just a thought, hope it gives you another perspective 

[PhilGUK]

Thank you for your replies and ideas.

Yes, we are on Viya, or at least some of our users are.  EG will be phased out, we are migrating existing projects. The organisation's priority was continuity, so lift and shift to compute, with access to CAS very limited at this point. It will come.  Anyway, we are seeing 8x speed improvement of Viya on new kit vs grid/Unix on 10 year old kit.  Whilst parallel processing in EG was a useful idea, most users will be happy with that upgrade.  There may be the odd code, especially the one I did get running in parallel, where we might want to keep it (hence I was asked about it).

We are very much a community of users (500+ active programmers), but without a central programming support (we have systems support, obviously) users are left to their own imagination and what we can get from each other.  Hence, most of our code starts at the beginning, goes through the middle and gets to the end and tends to be bespoke for each challenge we meet.

[PhilGUK]

Trying to do this in EG, I got less far that I had 2-3 years ago.  If I run it on SASApp, the signon gives me a load of errors.  If I run it on Local, the signon works and I can run code in Rsubmit to 3 different servers; but then I had a few problems mixing code running on Local and code running on SASApp.

Then I remembered, that the ask was "SAS Studio on Compute on Viya" I took some simple code into that environment and made some progress:
Signon worked.
Rsubmit para_i WAIT=YES; libname sWork "/sasvpath/.."; EndRsubmit; worked fine.
Control was passed back to the main server and I created a 10m obs, dataset:   data series_1; do i = 10000001 to 20000000; output; end; run;  Fine.
Ran the same code on each rSubmit; creating sWork.para_i;  this time with WAIT=NO (expecting they would run in parallel).
A WAITFOR _ALL_; before running some more code on the main server to append the datasets:  series_1 and sWork.para_1, sWork.para_2 and sWork.para_3; using data and SET.   This all worked, but the remote code was ridiculously slow and it wasn't clear what if anything was running in parallel.

From the log
Signon seemed to take a good time longer than the log showed (perhaps 8s lag each)
Creating work.series_1  took 0.27s.
The FULLSTIMER log between the first and 2nd "main" steps was 38s; during which time the rsubmits were running.
Only after the signoff did the log include the code that was in the RSubmit WAIT=NO steps.  They showed it took 12.6s, 18.4s and 18.6s to create the 10m obs para_i datasets.  This is a total of 48s, which is more than the 37s between the two "main" steps, but 37s is longer than the 19s of the longest remote.  

From those times, I don't know what is running in parallel, or why it is a 100x slower to write a 82Mb file on the perm folder in Rsubmit than in Work on the main connection; unless it is 18s of overhead + 0.3s of work.

I appreciate that I am trying to run something that is very quick on the Rsubmit, but when I accidentally ran it for 55m obs in the loop, it was taking forever, enough minutes for me to regret it and abort it.

Thanks for everyone's suggestions, but I am going to close this thread.  I will pass my attempt on to the programmer who asked about it, as they most likely have a genuine use case.  It also looks like we have some setup that I don't fully understand or have access to, so any solution looks like it needs some systems support.

I'll mark SASKiwi solution as best, even if I haven't been able to successfully produce useful results in my set up.

[Tom]

How long does it take for your VIYA instance to execute the SIGNON command to initiate a new "remote" session of SAS?

I am trying to figure out if the delays I am seeing is because of our installation or if it is just the nature of using VIYA instead of SAS.

On our new VIYA install a simple command like:

signon another sascmd='!SASCMD';

Takes over a minute to execute.

NOTE: Remote signon to ANOTHER complete.
100  %bench;
Elapsed seconds = 63.1328899860382

On our SAS 9.4 installation is takes a fraction of a second.

 NOTE: Remote signon to ANOTHER complete.
 75         %bench;
 Elapsed seconds = 0.30531001091003

[PhilGUK]

I hadn't put a timer on it, as it was just seeing if it worked at all, just it seemed to take longer for the log to build, that it claimed in the log to have taken.
and from the log

%put %sysfunc(time(), time8.);
signon para1;
%put %sysfunc(time(), time8.);
signoff _all_;

and from the log, the start time was 14:33:31 and the end time 14:34:13, so that was 42s.  The log showed initialisation of 5s real time and autoexec of 0.
So I don't know what is going on!.  I use SignOn when I have to run BASE.  Without testing that, I can assure you it takes next to no time.

14:33:31
81   signon para1;

NOTE: SAS initialization used:
      real time           5.08 seconds
      cpu time            0.05 seconds
      
NOTE: AUTOEXEC processing beginning; file is 
NOTE: DATA statement used (Total process time):
      real time           0.00 seconds
      cpu time            0.01 seconds
      
NOTE: AUTOEXEC processing completed.
NOTE: Remote signon to PARA1 complete.
82   %put %sysfunc(time(), time8.);
14:34:13
83   signoff _all_;

[PhilGUK]

@Tom I ran it again today.  This time, using just one remote signon.  That took about 25s to do the sign on.  For the 10m obs dataset written to sWork perm lib, it took 6s, rather than the 18s from yesterday.  Tried writing to WORK, and it matched the 0.2s runtime of the main server.  

Not a big experiment, but it appears our perm server for small (80Mb) files is far slower than to WORK.
If I run it to (work) in EG on Local (Windows), it took 0.3s,  and about 1.5s on our Unix box.

The colleague who asked me about this is now running with it.  Hopefully they have a long running set of jobs that write to work and only have to put the final datasets to perm sWork to share and they will be more successful (or if there is less overhead with large files to perm)