topic Re: how to run macros in parallel in SAS Programming

how to run macros in parallel

iscgonzalez — Mon, 14 Feb 2022 21:47:39 GMT

Hello,

I have a program that contains multiple proc sql on macros, something like this:

%macro m1000();

proc sql;

create table m1000 as

select * from test1000;

run;

%mend;

I have around 70 macros like this and my question is, how could I run those 70 macros having 5 processes in parallel at the same time to run them (like a queue with only 5 threads) and as soon as one finishes, start running the next. I explored the systask command, but couldn't achieve to do this.

Re: how to run macros in parallel

Tom — Mon, 14 Feb 2022 22:37:38 GMT

Not sure why you have phrased the question in terms of running macros. Macros are just a way to generate SAS code.

So what are the programs you want to run in parallel? What do they do? How are they related to each other? Are the all totally independent? If they are related how? Does the results of running one impact the others? Or are they just related by wanting to start from some common setup ? Or perhaps because you want to aggregate the results when they are done?

Do you have SAS/Connect licensed and installed?

Re: how to run macros in parallel

Reeza — Mon, 14 Feb 2022 22:50:14 GMT

How is your SAS installation set up?

Is it just Base SAS? EG? SAS Studio? Viya?

Is it a server installation or desktop?

To check if you have SAS CONNECT run the following:

proc product_status;run;

I'm assuming you've already perused the LexJansen site for papers on parallel processing?

https://www.lexjansen.com/search/searchresults.php?q=parallel%20processing%20sas

Re: how to run macros in parallel

ballardw — Mon, 14 Feb 2022 23:28:32 GMT

@iscgonzalez wrote:

Hello,

I have a program that contains multiple proc sql on macros, something like this:

%macro m1000();

proc sql;

create table m1000 as

select * from test1000;

run;

%mend;

I have around 70 macros like this and my question is, how could I run those 70 macros having 5 processes in parallel at the same time to run them (like a queue with only 5 threads) and as soon as one finishes, start running the next. I explored the systask command, but couldn't achieve to do this.

For that specific code you would be better off renaming the data set test1000 to m1000. It would be faster and if these sets are of any size, a whole lot faster, could be done in one call to proc data sets. Example:

data junk1;
   x=3;
run;
data junk2;
  x=4;
run;

proc datasets library=work nolist nodetails;
   change Junk1 = New1
          Junk2 = New2
   ;

run;
quit;

As an aside, the above approach won't let you accidentally overwrite an existing "new" data set if exists, which might be a good thing if this is getting turned loose on unsuspecting data.

Re: how to run macros in parallel

Sajid01 — Mon, 14 Feb 2022 23:38:32 GMT

Hello @iscgonzalez
As rightly pointed out by @Tom Macros are simply text replacement tokens.

from your question, I see that you want to run the 70 segments of code in parallel.
if they can be run in parallel then SAS grid can be a good candidate. One should also factor the I/O requirements.

Re: how to run macros in parallel

SASKiwi — Mon, 14 Feb 2022 23:46:20 GMT

If you run your programs in batch mode then you you can run as many as you like at the same time as long as each runs independently of each other. No need for fancy splitting or SYSTASK.

Re: how to run macros in parallel

sbxkoenk — Tue, 15 Feb 2022 00:43:35 GMT

Hello,

Many ways to run things in parallel.

My favorite V9.4 way is MP CONNECT. See below.

(Let me know if you want to set up // CAS sessions in SAS VIYA 3.5+ and I will post, in this topic thread, some interesting blogs on // CAS sessions).

Base SAS + SAS/CONNECT - A simple method to generate load on any number of licensed cores
Posted 04-08-2021 05:46 AM | by SimonWilliams
https://communities.sas.com/t5/SAS-Communities-Library/Base-SAS-SAS-CONNECT-A-simple-method-to-generate-load-on-any/ta-p/732174

Running SAS programs in parallel using SAS/CONNECT®
By Leonid Batkhan on SAS Users January 13, 2021
https://blogs.sas.com/content/sgf/2021/01/13/running-sas-programs-in-parallel-using-sas-connect/

Using SYSTASK and SAS macro loops for massively parallel processing
By Leonid Batkhan on SAS Users June 14, 2021
https://blogs.sas.com/content/sgf/2021/06/14/using-systask-and-sas-macro-loops-for-massively-parallel-processing/

Koen