Looking for LSF detailed flow information Linux

[sergie89]

Hi guys,

I am curious if there is a custom LSF (9.1) script  for flow statistics like Job name, Start time, End time, Total time, Status. Does anyone have experience with this?

[gwootton]

LSF and Process Manager have commands like bjobs/jflows and bhist/jhist for information about jobs ran or running in lsf (b*) and flows run from Process Manager (j*).

[Brandon]

We pipe the results of a jhist command to a file then parse the results into a SAS table.  Like Greg mentioned, there are commands to get just jobs or flow info.  jhist will return historical information about Process Manager Server, calendars, flow definitions, flows, and jobs.  We leverage -C to filter the results to just 'job,flow' then using a -t we can get results just for the time window from our last processing time.  Then using SAS Visual Analytics we provide insight into the processes running through the scheduler.

-

Brandon Kirk | Sr Manager, IT Software Development

[jklaverstijn]

If you want to collect the historical data from LSF jobs and flows you can read the files that LSF keeps. There are a few in the LSF directory (jobs) and the process manager directory (flows). The structure of these files is documented by IBM.

The ones of most interest are:

• .../lsf/work/sas_cluster/logdir/lsb.events
  • https://www.ibm.com/support/knowledgecenter/en/SSWRJV_10.1.0/lsf_config_ref/lsb.events.5.html
• .../lsf/work/sas_cluster/logdir/lsb.acct
  • https://www.ibm.com/support/knowledgecenter/en/SSWRJV_10.1.0/lsf_config_ref/lsb.acct.5.html
• pm/work/history/history.log.*

These files contain the data that you get from the bjobs, bhist and jhist commands. Depending on how you configured LSF these commands (and the Flowmanager application) show only recent history. This approach allows you to keep the info for an indefinite amount of time.

Let me also point you to the EOM Job Monitor that builds a database of your job and flow execution and has a effective flow-centric web interface.