topic Re: SAS Grid - Jobs appear with submitted date before restarting time in Administration and Deployment

SAS Grid - Jobs appear with submitted date before restarting time

MariaD — Thu, 23 Jan 2020 13:22:42 GMT

Hi Folks,

We have a SAS Grid installed on Linux. We restarted our server and after that we find few jobs (listed using bjobs command) with submitted date before the restarted time. How could be possible?

Regards,

Re: SAS Grid - Jobs appear with submitted date before restarting time

rbetancourt — Thu, 23 Jan 2020 18:56:17 GMT

I recommend you ensure all of the hot-fixes are up to date. I recently deployed 9.4M5 Grid cluster and recall there being a hotfix which addressed jobs executed with SASGSUB report the correct job start time to LSF.

Re: SAS Grid - Jobs appear with submitted date before restarting time

AnandVyas — Fri, 24 Jan 2020 05:55:40 GMT

Hi @MariaD,

Restarted server or SAS Services? LSF services also restarted? What is the state of these jobs? By any chance they were in PEND state while restarting?

Re: SAS Grid - Jobs appear with submitted date before restarting time

MariaD — Fri, 24 Jan 2020 09:36:35 GMT

Hi @AnandVyas ,

We restarted the entire server, not only the services. The state of the JOB is RUN.

Regards,

Re: SAS Grid - Jobs appear with submitted date before restarting time

doug_sas — Fri, 24 Jan 2020 12:26:37 GMT

Were the jobs submitted before or after the server was restarted? If jobs were submitted after the server restart and show the submit time before the restart something is wrong. If jobs were submitted before the restart, then everything is normal.

Jobs can be submitted, LSF restarted, and the jobs will still show the time they were submitted because LSF logs 'events' (like job submission/starting/termination) that it reads when it comes up or when a master changes.

Re: SAS Grid - Jobs appear with submitted date before restarting time

MariaD — Fri, 24 Jan 2020 12:32:35 GMT

Hi @doug_sas

The jobs were submitted before we restarted the server. One more question, which is the best way to kill a job preventing zombie or defunct process/jobs? Sometimes when we use bkill the jobs is not killed correctly and it became defunct or zombie. Even if we user bkill -9 it happens. Any way to prevent it?

Regards,

Re: SAS Grid - Jobs appear with submitted date before restarting time

doug_sas — Fri, 24 Jan 2020 12:45:38 GMT

If you just want to remove them from LSF, you can use 'bkill -r <jobID>'.

If they are removed from LSF, but the OS has not killed them the processes probably have not handled the SIGTERM/SIGKILL signal correctly.

You could talk to SAS tech support to see if IBM tech support understands why it is not working.

Re: SAS Grid - Jobs appear with submitted date before restarting time

MariaD — Fri, 31 Jan 2020 14:45:47 GMT

Hi @rbetancourt

Thanks, do you know the name/link of the hotfix?

Regards,

Re: SAS Grid - Jobs appear with submitted date before restarting time

rbetancourt — Fri, 31 Jan 2020 18:48:52 GMT

Hi @MariaD,

In our case, it was:

http://support.sas.com/kb/61/718.html

HTH.

Best regards,

Randy