topic Re: 10 jobs in the same time in Administration and Deployment

10 jobs in the same time

danielmitu — Fri, 29 Jun 2018 14:18:14 GMT

Hello,

Can you help me with one issue?

In this moment, on our system, in flow manager, there are 6 jobs that can run in the same time.

How can I increase those up to 10 simultaneous jobs ?

SAS 9.4 M2, Windows Server 2012 R2.

Thanks

Re: 10 jobs in the same time

JuanS_OCS — Fri, 29 Jun 2018 14:26:54 GMT

Hello @danielmitu,

you would need to check the number of cpus/threads your system can handle, and modify the size of your queues.

Just one reference, if you look into SAS or LSF documentation, or even just google, you will find several more:

https://www.ibm.com/support/knowledgecenter/en/SSWRJV_10.1.0/lsf_admin/queue_restrict_job_size.html

Re: 10 jobs in the same time

danielmitu — Fri, 29 Jun 2018 14:37:05 GMT

Hello,

You are saying that if I have 6 cpus/cores I can have only 1 job/cpu ?

Thanks

Re: 10 jobs in the same time

JuanS_OCS — Fri, 29 Jun 2018 14:48:07 GMT

Well, let me explain: first of all, 1 core is not necessarily the same as 1 cpu. 1 core can handle 2 (real time) threads, hence it can count as 2 logical cpus. This is named as Hyper-threading.

Second, in real time, yes, a computer can handle only as many real time threads as cpus. In total, not only SAS. This means that also operating system and other services will be launching threads.

In user/logical time, the computer can (and does) handle thousands of threads, as better as it can, giving processor time (cpu) to the threads that need it. That is why user time for processes is longer than real time. Because processes have to wait.

And as such, the operating system has its own queues to distribute the threads along the cpus. LSF a Resource Manager that is just another layer of thread distribution that can provide better handling of jobs as execution units, and it has its own queues.

As you see, the queue is even another (logical) layer, that helps the resources manager to better grant resources to jobs. And jobs that come afterwards, they need to wait.

You can modify the queue lengths (logical limits) in LSF, just beware of the physical limitations.

Re: 10 jobs in the same time

danielmitu — Mon, 02 Jul 2018 08:51:23 GMT

And where I need to modify this limit ?

Thanks

Re: 10 jobs in the same time

SASKiwi — Tue, 03 Jul 2018 04:13:20 GMT

In this folder: SAS\Thirdparty\LSF\conf\lsbatch\sas_cluster\configdir you should find a file called lsb.hosts.

In the file you should find a section that looks like this. The MXJ column specifies the maximum number of jobs per host. Note in this example a maximum of 8 jobs per host is specified.

Begin Host
HOST_NAME MXJ r1m pg ls tmp DISPATCH_WINDOW # Keywords
#hostA () 3.5/4.5 15/ 12/15 0 () # Example
#hostB ! 3.5 15/18 12/ 0/ (5:19:00-1:8:30 20:00-8:30)
#hostC 1 3.5/5 18 15 () () # Example
#hostD ! () () () () () # Example
#hostE 4 () () () () () # Example
#SPARCIPC () 4.0/5.0 18 16 () () # Example
default 8 () () () () () # Example
End Host

Re: 10 jobs in the same time

danielmitu — Tue, 03 Jul 2018 06:19:44 GMT

Hello SASKiwi,

in my lsb.hosts file I have this:

Begin Host
HOST_NAME MXJ r1m pg ls tmp DISPATCH_WINDOW # Keywords
#hostA () 3.5/4.5 15/ 12/15 0 () # Example
#hostB ! 3.5 15/18 12/ 0/ (5:19:00-1:8:30 20:00-8:30)
#hostC 1 3.5/5 18 15 () () # Example
#hostD ! () () () () () # Example
#hostE 4 () () () () () # Example
#SPARCIPC () 4.0/5.0 18 16 () () # Example
default ! () () () () () # Example
End Host

What does "!" means ?

Thanks

Re: 10 jobs in the same time

JuanS_OCS — Thu, 05 Jul 2018 07:39:12 GMT

Hi @danielmitu,

https://www.ibm.com/support/knowledgecenter/en/SSETD4_9.1.2/lsf_config_ref/lsb.hosts.5.html

Use ! to make the number of job slots equal to the number of CPUs on a host.