The jobs are in waiting state in LSF

[Hema_12]

Hi All,

I have created some jobs in DI and deployed and create flow to schedule on platform process manager server(LSF), while triggering the jobs from SAS SMC its successfully schedule but the jobs are not running on lsf and not able to create logs for it.

I have restarted the services for LSF but still the issue is remain and i have checked flow in LSF so all the jobs in waiting state for prolong time.

I have checked logs for the lsf services and in this log file "mbatchd.log."  found below error:--

---------------------------------------------------------------------------------------------------------------------------------

Master daemon on host <server> dying; fatal error - see above messages for reason
Nov 13 19:46:26 2019 690 3 9.1.3 init_log: Mode <40777> not allowed for job log directory </opt/sas94/pss/lsf/work/sas_cluster1/logdir>; the permission bits for others should be 05

Please help me out.

Thanks & regards,

Hema

[JosvanderVelden]

Try this:
1) Check the logdir directory permission, please change ownership to lsfadmin and set the permission 755 (others should be 5).
2) Restart the lsf and jdf services.

[Tom]

Our admin has setup many queues in LSF and we had a problem recently where one of the them was dependent on a particular execution node for SAS.  But they had shut down that node for some reason. So all of the jobs in that queue just sat waiting because LSF could not find a place to run them.

Check and make sure you don't have a similar resource availability issue.

[Anand_V]

Hi @Hema_12,

Are you facing this issue recently? The timestamp of the log seems to be old. Can you long list the job using -l flag with bjobs and check if any info on why the job is waiting displayed?