SAS Data Integration Studio, DataFlux Data Management Studio, SAS/ACCESS, SAS Data Loader for Hadoop and others

how can we run many jobs concurrently using Data Integration?

Accepted Solution Solved
Reply
Occasional Contributor
Posts: 14
Accepted Solution

how can we run many jobs concurrently using Data Integration?


Dear,

how can we run many jobs concurrently using Data Integration? and is there any max number of Jobs to execute each time?


Accepted Solutions
Solution
‎12-28-2012 07:47 PM
Respected Advisor
Posts: 4,173

Re: how can we run many jobs concurrently using Data Integration?

Posted in reply to MS_Egyptian

I agree with Barry that it's the scheduler where you set up how jobs are run.

Within DIS you can use the loop transformation which allows you to execute the inner jobs in parallel, you can set the maximum number of jobs to be executing at the same time and you can also select to run these jobs on a grid.

As great as the loop transformation is to run a parameterised inner job it also has the disadvantage that from a scheduler perspective only the outer job is monitored. So everything runs as one big junk and if one of the inner jobs fails it's up to you to figure out which one failed (the outer job will get an error code which then is passed to the scheduler so you see there the outer job failing).

I would expect Control M to give you an option for controlling how many jobs are run in parallel. Unlike LSF Control M is not integrated into SAS Metadata so you will have to build the scheduling outside of SAS.

View solution in original post


All Replies
Respected Advisor
Posts: 4,173

Re: how can we run many jobs concurrently using Data Integration?

Posted in reply to MS_Egyptian

SAS version? DIS version? OS? What scheduler are you using?

Occasional Contributor
Posts: 14

Re: how can we run many jobs concurrently using Data Integration?

SAS Version  : 9.3,

DIS version    : 4.5

OS                   : SunOS SASTEST 5.10 Generic_147440-15 sun4v sparc sun4v

 

Scheduler       : Automatic Job Scheduler (Control M Enterprise Manager)

thanx

Super Contributor
Posts: 356

Re: how can we run many jobs concurrently using Data Integration?

Posted in reply to MS_Egyptian

I would have thought it was the scheduler that controled this not Data intergration (assuming the jobs have been scheduled) - In LSF the default I think is paired to the number of CPU's but can be changed in a LSF config file

Solution
‎12-28-2012 07:47 PM
Respected Advisor
Posts: 4,173

Re: how can we run many jobs concurrently using Data Integration?

Posted in reply to MS_Egyptian

I agree with Barry that it's the scheduler where you set up how jobs are run.

Within DIS you can use the loop transformation which allows you to execute the inner jobs in parallel, you can set the maximum number of jobs to be executing at the same time and you can also select to run these jobs on a grid.

As great as the loop transformation is to run a parameterised inner job it also has the disadvantage that from a scheduler perspective only the outer job is monitored. So everything runs as one big junk and if one of the inner jobs fails it's up to you to figure out which one failed (the outer job will get an error code which then is passed to the scheduler so you see there the outer job failing).

I would expect Control M to give you an option for controlling how many jobs are run in parallel. Unlike LSF Control M is not integrated into SAS Metadata so you will have to build the scheduling outside of SAS.

Occasional Contributor
Posts: 14

Re: how can we run many jobs concurrently using Data Integration?

Thanks for both Barry and Patrick

🔒 This topic is solved and locked.

Need further help from the community? Please ask a new question.

Discussion stats
  • 5 replies
  • 981 views
  • 3 likes
  • 3 in conversation