topic Re: Paralle Iterator or Fork for Parallel Processing in New SAS User

Paralle Iterator or Fork for Parallel Processing

Shakti_Sourav — Tue, 12 Jul 2022 07:15:31 GMT

Dear Team,

I have created a Data job having 30 nodes and it's taking 16 hours to complete. I want to run the job in Parallel. How to do that ?

I have attached one data job which is conatins 30 nodes.

I have created New process job and then what will I do ?

Thanks and regrads,

Shakti

Re: Paralle Iterator or Fork for Parallel Processing

JosvanderVelden — Tue, 12 Jul 2022 08:35:22 GMT

Can you provide some more details?
For instance:
Is the job created with data management studio or another SAS product/solution?
Where will the job run? A single server? A cluster? How many sas workspaces do you have?
Is all data in SAS or do you have a RDBMS or cloud storage involved?

Re: Paralle Iterator or Fork for Parallel Processing

Shakti_Sourav — Tue, 12 Jul 2022 09:44:43 GMT

Hi
1. Job is created in the Data management Studio.
2. Job run in DM server
3. I have connected to the Oracle.

Re: Paralle Iterator or Fork for Parallel Processing

JosvanderVelden — Thu, 14 Jul 2022 09:12:19 GMT

Have you seen this thread? https://communities.sas.com/t5/SAS-Data-Management/Parallel-Data-Jobs-in-a-Process-Job/td-p/148824

Can all the flows you have run in parallel or are there steps that are dependent on other steps?

Re: Paralle Iterator or Fork for Parallel Processing

SASKiwi — Thu, 14 Jul 2022 21:02:57 GMT

The way I run SAS DQ jobs in parallel is to split the DQ job into separate jobs, one for each parallel stream. Then I run the DQ jobs from a SAS program which uses SAS/CONNECT to run the jobs in separate SAS sessions. If you don't have SAS/CONNECT you could always just run separate SAS programs as batch jobs at the same time. The job originally took over 12 hours and now runs in less than 4 hours.

Re: Paralle Iterator or Fork for Parallel Processing

Shakti_Sourav — Fri, 15 Jul 2022 02:56:58 GMT

I have created Process job. Then I took Parallel Iterator node for parallel processing. Under Parallel processing, I referred my batch job based on Path.

Is it way to parallel processing or what ?

Re: Paralle Iterator or Fork for Parallel Processing

SASKiwi — Fri, 15 Jul 2022 03:15:17 GMT

I've never tried it the way you are doing it. I call a DQ job from a SAS program like this:

data _null_;
  rc        = dmsrvauthdom("DMServerAuth");
  JobName   = 'My DQ Job Name.ddf';
  jobid     = dmsrvBatchJob (JobName, 'http://MyDQAppServer.com', 21036);
  jobstatus = dmsrvJobStatus(jobid, 'http://MyDQAppServer.com', 21036, -1, 5);
  put _all_;
run;

If you are not familiar with this way and you don't know how to code SAS programs then I suggest that you continue with your way and open a Tech Support track to get help.

Re: Paralle Iterator or Fork for Parallel Processing

SASKiwi — Fri, 15 Jul 2022 04:48:19 GMT

After a bit of research I think you need to use the Fork node. The Parallel Iterator is for when you want to loop multiple times in parallel.