BookmarkSubscribeRSS Feed
🔒 This topic is solved and locked. Need further help from the community? Please sign in and ask a new question.
MS_Egyptian
Fluorite | Level 6


Dear,

how can we run many jobs concurrently using Data Integration? and is there any max number of Jobs to execute each time?

1 ACCEPTED SOLUTION

Accepted Solutions
Patrick
Opal | Level 21

I agree with Barry that it's the scheduler where you set up how jobs are run.

Within DIS you can use the loop transformation which allows you to execute the inner jobs in parallel, you can set the maximum number of jobs to be executing at the same time and you can also select to run these jobs on a grid.

As great as the loop transformation is to run a parameterised inner job it also has the disadvantage that from a scheduler perspective only the outer job is monitored. So everything runs as one big junk and if one of the inner jobs fails it's up to you to figure out which one failed (the outer job will get an error code which then is passed to the scheduler so you see there the outer job failing).

I would expect Control M to give you an option for controlling how many jobs are run in parallel. Unlike LSF Control M is not integrated into SAS Metadata so you will have to build the scheduling outside of SAS.

View solution in original post

5 REPLIES 5
Patrick
Opal | Level 21

SAS version? DIS version? OS? What scheduler are you using?

MS_Egyptian
Fluorite | Level 6

SAS Version  : 9.3,

DIS version    : 4.5

OS                   : SunOS SASTEST 5.10 Generic_147440-15 sun4v sparc sun4v

 

Scheduler       : Automatic Job Scheduler (Control M Enterprise Manager)

thanx

twocanbazza
Quartz | Level 8

I would have thought it was the scheduler that controled this not Data intergration (assuming the jobs have been scheduled) - In LSF the default I think is paired to the number of CPU's but can be changed in a LSF config file

Patrick
Opal | Level 21

I agree with Barry that it's the scheduler where you set up how jobs are run.

Within DIS you can use the loop transformation which allows you to execute the inner jobs in parallel, you can set the maximum number of jobs to be executing at the same time and you can also select to run these jobs on a grid.

As great as the loop transformation is to run a parameterised inner job it also has the disadvantage that from a scheduler perspective only the outer job is monitored. So everything runs as one big junk and if one of the inner jobs fails it's up to you to figure out which one failed (the outer job will get an error code which then is passed to the scheduler so you see there the outer job failing).

I would expect Control M to give you an option for controlling how many jobs are run in parallel. Unlike LSF Control M is not integrated into SAS Metadata so you will have to build the scheduling outside of SAS.

MS_Egyptian
Fluorite | Level 6

Thanks for both Barry and Patrick

sas-innovate-2024.png

Join us for SAS Innovate April 16-19 at the Aria in Las Vegas. Bring the team and save big with our group pricing for a limited time only.

Pre-conference courses and tutorials are filling up fast and are always a sellout. Register today to reserve your seat.

 

Register now!

How to connect to databases in SAS Viya

Need to connect to databases in SAS Viya? SAS’ David Ghan shows you two methods – via SAS/ACCESS LIBNAME and SAS Data Connector SASLIBS – in this video.

Find more tutorials on the SAS Users YouTube channel.

Discussion stats
  • 5 replies
  • 2443 views
  • 3 likes
  • 3 in conversation