Full LSF and SAS Grid

[bdoug]

We have a full LSF license and SAS grid manager licensed but we are having an issue after the install & config with validating the SAS Grid Server.

Questions:

1. I've heard, but want to confirm that SAS grid manager can work correctly with a Full LSF License (license for LSF from IBM for a large cluster).  Can someone confirm this?
2. We installed SAS grid manager and the grid control server is on a host (Linux Server) that has LSF installed as an LSF Client.  When we try to validate the grid server from SAS management console it fails. The log contains these lines (cleaned up):

         %put rc=%sysfunc(grdsvc_enable(_all_,server="SASApp"));signon t1; rsubmit;proc options;run;endrsubmit;signoff _all_;

         rc=0

         NOTE: Remote session ID T1 will use the grid service _ALL_.

         NOTE: Remote signon to T1 commencing (SAS Release 9.04.01M3P062415).

         ERROR: PROVIDER(Platform): LIM is down; try later
         ERROR: Cannot initialize the grid provider module.
         ERROR: A communication subsystem partner link setup request failure has occurred.
         ERROR: Cannot initialize the grid provider module.
         ERROR: Remote signon to T1 canceled.
         ERROR: Invalid or unspecified remote session ID. Set OPTIONS REMOTE=session_id.
         ...  

I know LIM is up on the servers. 

I will try running a bsub sleep from that host later today, but meanwhile does anyone have any suggestions?

Thanks,

Bryan

[LinusH]

1. Yes, this is what I heard as well. But I don't think that this configuration is not fully supported by SAS. But my guess is that they would be helpful anyhow.

2. Grids are quite complex environments which makes them a bit tricky to trouble shoot over a forum. I sugest that you open a track to tech support.

[Timmy2383]

What about the res and sbatchd daemons?  

I would run the following command as lsfadmin on each grid node to make sure they are up "lsadmin limstartup; lsadmin resstartup; badmin hstartup".

Once you've confirmed they are up on all nodes, can you confirm that the nodes have been added as resources in the lsf.cluster file?

I think the file is typically named lsf.cluster.<your cluster name>, so mine is called "lsf.cluster.sas_cluster" under <LSF install dir>/conf.

Here's the section of my config file:

My file is configured to show all servers in our grid, but only the servers that are elligible to receive and process SAS jobs would have something other than the default "mg" under the "RESOURCES" column.  You just add the app server names.  So for me, only computenode01-07 can receive SAS jobs from LSF, for both my SASApp and SASEM app servers.

Hope this helps. Let me know what you find out.

[JuanS_OCS · Accepted solution]

Hi,

even if you have the full license, it is possible that the license files have not been updated properly during the installation. Therefore:

1 Stop Grid, Process manager, and all Platform LIM, etc.

2. Check that you can find a license.dat on the expected locations (basically, on the Process Manager, LSF and Shared_LSF  "conf" directories. If you find some under "old" directories, check them too. If not present, create as the others.

3. Check the content on each, its permissions, and check that the license is OK on each.

4. When ready, start the processes on the right order. 

5. Check with bjobs, lshosts, etc

6. Validate your servers through SMC, all.: You might need to restart all the Object Spawners if they cannot validate.

Hope this will help!

Regards, Juan