I am not a Linux admin type, but googling shows   sudo lsof -i -P -n | grep LISTEN or   netstat -tulpn | grep LISTEN   and look for the ports defined in lsf.conf. The defaults would be something like # Other variables LSF_LIM_PORT=7869 LSF_RES_PORT=6878 LSB_MBD_PORT=6881 LSB_SBD_PORT=6882 # Enable mbd query child LSB_QUERY_PORT=6891   ... View more

You cannot start LSF daemons using ports that are already in use by something else. Also make sure the ports are open through any firewalls between the nodes in the grid. ... View more

Is something already running and using the LSF ports defined in the lsf.conf file? ... View more

Is the shared directory mounted in the same location on all nodes? It should be to work. You cannot run lsadmin and badmin on the newly added node because that node is not a part of the grid yet so the command would be rejected. That is why it worked on the master node.   The host setup command should have started the daemons on the node. Check to make sure the daemons are running on the node and errors are not showing up in the <LSF_TOP>/log directory.   Hopefully the nodes can easily talk to one another using their names. ... View more

Is the path to the share the same on all nodes? Can you see the node in the lshosts, lsload, and bhosts commands run from the master? ... View more

PROC SCAPROC will generate code that tries to use SAS/CONNECT grid-enabled SIGNONs. When it runs, it will do a good job of sending the next task to an available grid server, but if that grid server dies, the RSUBMIT'd code is lost and no recovery is attempted.   It should be noted that PROC SCAPROC and the Distribute macro have different purposes: The Distribute macro (found in the Grid Toolbox) is meant to manage the SIGNON/RSUBMIT of independent 'tasks' to one or more CONNECT servers (usually grid servers, but it will spawn servers locally too). It requires code to be written in a specific form (called a 'task') that may be difficult to accomplish for some processing. It is best where the task being executed is the same on multiple servers with just data or parameters being different from one task to the next. Since everything is an independent task, it has the ability to restart a task if the host dies and the task processing fails.   PROC SCAPROC is meant to run SAS programs to analyze data access/ data processing patterns and try to make rewrite the code to make it run in parallel. It will insert macros that use SAS/CONNECT's grid-enabled SIGNON/RSUBMIT when data access/ data processing can be done in parallel. It will get code most of the way to parallel processing, but can be fooled by heavy use of macros that hide data access / data processing. Because the original SAS program is sequential, the parallel processing done by the output of PROC SCAPROC must be executed in a specific order which may not lend itself to retrying. ... View more

Checkpoint/restart of SAS code via the grid is only supported through batch processing via SASGSUB. SASGSUB's GRIDRESTARTOK (for data-step checkpoint restart) and GRIDLRESTARTOK (for label checkpoint restart) cause the batch job to put its checkpoint/restart information onto a shared directory so when the job restarts on another machine, it can find the checkpoint information.   EG creates an interactive connection to a grid-launched workspace server. The EG user may create SAS code that uses grid-enabled SIGNONs to take advantage of grid hosts for parallel processing, but those connections are also interactive. When code is submitted via EG, it runs on the workspace server - no new grid job is started to process the SAS job. Interactive grid jobs like grid-launched workspace servers and grid-launched CONNECT servers (a.k.a., grid-servers) cannot use checkpoint restart because once the connection is lost so is all of the data. There is also the Distribute macro that can manage individual tasks out onto grid servers via CONNECT SIGNON/RSUBMITs. It manages the distribution of tasks to available grid servers and will restart a task if the host it is currently running on dies.   ... View more

If you just want to remove them from LSF, you can use 'bkill -r <jobID>'.  If they are removed from LSF, but the OS has not killed them the processes probably have not handled the SIGTERM/SIGKILL signal correctly.   You could talk to SAS tech support to see if IBM tech support understands why it is not working. ... View more

Were the jobs submitted before or after the server was restarted? If jobs were submitted after the server restart and show the submit time before the restart something is wrong. If jobs were submitted before the restart, then everything is normal. Jobs can be submitted, LSF restarted, and the jobs will still show the time they were submitted because LSF logs 'events' (like job submission/starting/termination) that it reads when it comes up or when a master changes. ... View more