topic Re: Grid/LSF will not dispatch jobs in Administration and Deployment

Grid/LSF will not dispatch jobs

bdoug — Fri, 06 Jan 2017 16:29:32 GMT

Something is up with our SAS Grid (9.4M3). It is a new installation and even a simple "bsub sleep 20" request remains in PEND status.

bjobs -l 768

PENDING REASONS:
New job is waiting for scheduling;

SCHEDULING PARAMETERS:
r15s r1m r15m ut pg io ls it tmp swp mem
loadSched - - - - - - - - - - -
loadStop - - - - - - - - - - -

RESOURCE REQUIREMENT DETAILS:
Combined: -
Effective: -

Is there an LSF command line command that will give detailed information on why the job is not being dispatche, like the criteria LSF is using to keep the job in the pending status?

Re: Grid/LSF will not dispatch jobs

SASKiwi — Mon, 09 Jan 2017 08:13:28 GMT

This may be a long shot but this problem sounds very similar to one we struck. There is a weird bug in LSF at least up to V9.1 which happens when you configure it to send emails and an email error occurs. LSF creates a file called PROGRAM and stores it in the same root folder where LSF is installed (eg on Windows C:\Program Files) and writes the email SMTP error in it.

This rogue file called PROGRAM then blocks any use of LSF thereafter as it is executed instead of the real LSF program! Simply renaming the file will fix the problem.

Re: Grid/LSF will not dispatch jobs

bdoug — Wed, 11 Jan 2017 14:57:45 GMT

That is interesting, but not our issue. It turns out we had a NTP issue on our servers, which confused LSF.

Still wonder is there is a command to get LSF to tell you why it is not dispatching a job.

Re: Grid/LSF will not dispatch jobs

JuanS_OCS — Wed, 11 Jan 2017 16:18:33 GMT

Hello @bdoug,

I wonder if this information would help you out: http://www-01.ibm.com/support/docview.wss?uid=isg3T1016430

Besides this some items from my personal experience:

- I would check at IBM site and asking SAS Technical Support if there is any recommended patch available for your LSF version... normally there is something.

- Also on a Linux installation, I found this several times because some part of the installation and configuration was not done properly (normally, on the LSF config or pre-requisites).

- Please check also your queues config... it might be that the job is not going to the queue you expect, or just hanging there forever.

So I would propose: while you investigate this on your side or with us, I would contact SAS Technical Support, normally they have nice insights.

Best regard,

Juan

Re: Grid/LSF will not dispatch jobs

Kurt_Bremser — Thu, 12 Jan 2017 08:16:01 GMT

@bdoug wrote:

That is interesting, but not our issue. It turns out we had a NTP issue on our servers, which confused LSF.

Still wonder is there is a command to get LSF to tell you why it is not dispatching a job.

How should a dispatcher do this when the server's time is not correctly set? If it knew it should run a job, it would run the job. If the time is not right for the job, then the job shall not be run anyway, and no message needs to be sent.

If you had the dispatcher send you a message for every job it does not run at the moment for any reason, you'd be drowned in messages in seconds.

Re: Grid/LSF will not dispatch jobs

bdoug — Mon, 12 Jun 2017 15:48:12 GMT

Maybe instead of sending a message, supply the reason for pending in extended bjobs information? bjobs -l <ID>

Therefore, I can see why a job is pending but only when I ask.