BookmarkSubscribeRSS Feed
🔒 This topic is solved and locked. Need further help from the community? Please sign in and ask a new question.
bdoug
Obsidian | Level 7

We have a full LSF license and SAS grid manager licensed but we are having an issue after the install & config with validating the SAS Grid Server.

 

Questions:

  1. I've heard, but want to confirm that SAS grid manager can work correctly with a Full LSF License (license for LSF from IBM for a large cluster).  Can someone confirm this?
  2. We installed SAS grid manager and the grid control server is on a host (Linux Server) that has LSF installed as an LSF Client.  When we try to validate the grid server from SAS management console it fails. The log contains these lines (cleaned up):

         %put rc=%sysfunc(grdsvc_enable(_all_,server="SASApp"));signon t1; rsubmit;proc options;run;endrsubmit;signoff _all_;

         rc=0

         NOTE: Remote session ID T1 will use the grid service _ALL_.

         NOTE: Remote signon to T1 commencing (SAS Release 9.04.01M3P062415).

         ERROR: PROVIDER(Platform): LIM is down; try later
         ERROR: Cannot initialize the grid provider module.
         ERROR: A communication subsystem partner link setup request failure has occurred.
         ERROR: Cannot initialize the grid provider module.
         ERROR: Remote signon to T1 canceled.
         ERROR: Invalid or unspecified remote session ID. Set OPTIONS REMOTE=session_id.
         ...  

 

I know LIM is up on the servers. 

 

I will try running a bsub sleep from that host later today, but meanwhile does anyone have any suggestions?

 

Thanks,

 

Bryan

1 ACCEPTED SOLUTION

Accepted Solutions
JuanS_OCS
Amethyst | Level 16

Hi,

 

even if you have the full license, it is possible that the license files have not been updated properly during the installation. Therefore:

 

1 Stop Grid, Process manager, and all Platform LIM, etc.

2. Check that you can find a license.dat on the expected locations (basically, on the Process Manager, LSF and Shared_LSF  "conf" directories. If you find some under "old" directories, check them too. If not present, create as the others.

3. Check the content on each, its permissions, and check that the license is OK on each.

4. When ready, start the processes on the right order. 

5. Check with bjobs, lshosts, etc

6. Validate your servers through SMC, all.: You might need to restart all the Object Spawners if they cannot validate.

 

Hope this will help!

 

Regards, Juan

View solution in original post

3 REPLIES 3
LinusH
Tourmaline | Level 20
1. Yes, this is what I heard as well. But I don't think that this configuration is not fully supported by SAS. But my guess is that they would be helpful anyhow.

2. Grids are quite complex environments which makes them a bit tricky to trouble shoot over a forum. I sugest that you open a track to tech support.
Data never sleeps
Timmy2383
Lapis Lazuli | Level 10

What about the res and sbatchd daemons?  

 

I would run the following command as lsfadmin on each grid node to make sure they are up "lsadmin limstartup; lsadmin resstartup; badmin hstartup".

 

Once you've confirmed they are up on all nodes, can you confirm that the nodes have been added as resources in the lsf.cluster file?

 

I think the file is typically named lsf.cluster.<your cluster name>, so mine is called "lsf.cluster.sas_cluster" under <LSF install dir>/conf.

 

Here's the section of my config file:

 

Capture.PNG

 

My file is configured to show all servers in our grid, but only the servers that are elligible to receive and process SAS jobs would have something other than the default "mg" under the "RESOURCES" column.  You just add the app server names.  So for me, only computenode01-07 can receive SAS jobs from LSF, for both my SASApp and SASEM app servers.

 

Hope this helps. Let me know what you find out.

 

JuanS_OCS
Amethyst | Level 16

Hi,

 

even if you have the full license, it is possible that the license files have not been updated properly during the installation. Therefore:

 

1 Stop Grid, Process manager, and all Platform LIM, etc.

2. Check that you can find a license.dat on the expected locations (basically, on the Process Manager, LSF and Shared_LSF  "conf" directories. If you find some under "old" directories, check them too. If not present, create as the others.

3. Check the content on each, its permissions, and check that the license is OK on each.

4. When ready, start the processes on the right order. 

5. Check with bjobs, lshosts, etc

6. Validate your servers through SMC, all.: You might need to restart all the Object Spawners if they cannot validate.

 

Hope this will help!

 

Regards, Juan

suga badge.PNGThe SAS Users Group for Administrators (SUGA) is open to all SAS administrators and architects who install, update, manage or maintain a SAS deployment. 

Join SUGA 

CLI in SAS Viya

Learn how to install the SAS Viya CLI and a few commands you may find useful in this video by SAS’ Darrell Barton.

Find more tutorials on the SAS Users YouTube channel.

Discussion stats
  • 3 replies
  • 4606 views
  • 1 like
  • 4 in conversation