Please make sure the LASR server has been started with the LOGGING option. Also, let's enable TKMPI_DEBUGONEXCEPTION as described below to see if there are any exceptions. Note below that gstack will also work the same as pstack. pstack is a symlink to gstack in modern versions of GDB. Check that pstack is installed on all TKGrid nodes, by running 'pstack' at the command line. If it is, create a file called pstack.sh, which contains:
#! /bin/sh
pstack $2 > /tmp/sastb.out
Make the script executable.
Set variables like the following in the /<PATH_TO>/TKGrid/tkmpirsh.sh file on all TKGrid nodes:
export TKMPI_DEBUGGER=/opt/sas/pstack.sh
export TKMPI_DEBUGONEXCEPTION=1
After the LASR crash, check /tmp directory on all the machines. One or more may have a traceback in /tmp/sastb.out. Send this file wherever you find it. Also, I would like to review the LASR server logs and /var/log/messages file from all TKGrid nodes. In addition to all, I said above, show me the output from these commands:
/<SASHome>/SASFoundation/9.4/utilities/bin/vercon /<SASHome>/SASFoundation/9.4/sasexe/lasr.so
ls -ld /<SASHome>/SASFoundation/9.4/sasexe/lasr.so
/<SASHome>/SASFoundation/9.4/utilities/bin/vercon /<PATH_TO>/TKGrid/lib/lasr.so
/<PATH_TO>/TKGrid/bin/simsh "ls -ld /<PATH_TO>/TKGrid/lib/lasr.so"
