Public LASR server goes down repeatedly

[sasprofile]

Hello friends 

Can anyone please answer what are the reasons for Public LASR server going down repeatedly.

our public lasr server going down daily and we have to start it manually.

any suggestions how to trouble shoot this issue and what are different locations to look for the logs.

I could not find anything in /var/log/messages and webappserver on Midtier and VA servers, any other location  i need to look for the root cause of the issue. would really appreciate for your valuable advices.

Thanks all in advance

[alexal]

How many LASR servers do you have? All of them are going down? Do they run in a distributed or non-distributed mode? I also would like to know the exact SAS version.

[sasprofile]

Its Distributed, 7.4

[alexal]

Please make sure the LASR server has been started with the LOGGING option. Also, let's enable TKMPI_DEBUGONEXCEPTION as described below to see if there are any exceptions. Note below that gstack will also work the same as pstack.  pstack is a symlink to gstack in modern versions of GDB. Check that pstack is installed on all TKGrid nodes, by running 'pstack' at the command line.  If it is, create a file called pstack.sh, which contains:
 

#! /bin/sh
pstack $2 > /tmp/sastb.out

 Make the script executable.
 
Set variables like the following in the /<PATH_TO>/TKGrid/tkmpirsh.sh file on all TKGrid nodes:
 

export TKMPI_DEBUGGER=/opt/sas/pstack.sh
export TKMPI_DEBUGONEXCEPTION=1

After the LASR crash, check /tmp directory on all the machines. One or more may have a traceback in /tmp/sastb.out. Send this file wherever you find it. Also, I would like to review the LASR server logs and /var/log/messages file from all TKGrid nodes. In addition to all, I said above, show me the output from these commands:

/<SASHome>/SASFoundation/9.4/utilities/bin/vercon /<SASHome>/SASFoundation/9.4/sasexe/lasr.so
ls -ld /<SASHome>/SASFoundation/9.4/sasexe/lasr.so
/<SASHome>/SASFoundation/9.4/utilities/bin/vercon /<PATH_TO>/TKGrid/lib/lasr.so
/<PATH_TO>/TKGrid/bin/simsh "ls -ld /<PATH_TO>/TKGrid/lib/lasr.so"