SAS Web App Server unable to start

[MariaD]

Hi Folks, 

We have a Linux server running SAS 9.4M7. We applied the new license renewal 10 days ago on our server. On Monday, when the user tried to open SAS Visual Analytics appears 404 error. After that error, we restarted all SAS services. We checked that no were pending processes before starting the services. All services started ok except for SAS Web App Server (Server1_1, Server11_1, Server12_1, Server2_1, Server7_1, Server8_1). 

We even restarted the server physically, but the same situation occurred. We can't find any error in the logs.

We have the same server, on Test environment. On that environment, we didn't apply the license yet. So, we applied the license on it and restarted the services. To our surprise, the same error occurs. The SAS Web App Server was unable to start. All other services, like JMS, Cache Locator, Web Server, and Environment Manager appear to be fine.

Any ideas or suggestions? We already contact Tech Support, but until now we can't solve it and we don't have the SAS available. 

Regards, 

[Nigel_Pain]

Hi Maria

Just to ask the obvious question, have you updated the SID in the metadata? This is something you need to do as well as renewing the licence, and covers the middle-tier products. It's an option in SAS Deployment Manager:

[MariaD]

Hi @Nigel_Pain ,

Yes, we updated the SID file in the metadata too. 

Regards, 

[411752]

What does your Webserver log and WebApp server(SAServer1_1) log say, You should get a good clue looking at these both logs. Please attach them here ,if possible.

[MariaD]

Hi @411752 , 

I've just looked again on .../Lev1/Web/WebAppServer/SASServer1_1/logs and there isn't any new log generated after restarting the services. 

On ../Lev1/Web/WebServer/logs, the log called error_2023-03-17-12.13.log only have the lines below:

[Fri Mar 17 12:13:46.962498 2023] [mpm_event:notice] [pid 21601:tid 139623033902912] AH00489: Apache/2.4.43 (Unix) mod_bmx/0.9.6 OpenSSL/1.0.2u-fips configured -- resuming normal operations
[Fri Mar 17 12:13:46.962619 2023] [core:notice] [pid 21601:tid 139623033902912] AH00094: Command line: '/sas/software/SASWebServer/9.4/httpd-2.4/bin/httpd -d /sas/config/Lev1/Web/WebServer'

If you need any other log, please let me know. 

[StefanM]

Did you try it directly?
/sso/biconfig/940/Lev1/Web/WebServer/bin/httpdctl status|start

[MariaD]

Hi @StefanM ,

Yes, the WebServer is running.

Regards, 

[StefanM]

And the same with tomcat (Server1)?

/sso/biconfig/940/Lev1/Web/WebAppServer/SASServer1_1/bin/tcruntime-ctl.sh status|start

[411752]

@MariaD If all shows running fine and no errors on logs, I doubt some issues with Cache locator (on your primary compute and primary midtier) . You need to stop Cache locator on primary grid and stop all services on midtier nodes and restart.

Stop Cache locator on both nodes and delete below file and directory /opt/sas/config/Lev1/Web/gemfire/instances/ins_41415/ConfigDiskDir_

/opt/sas/config/Lev1/Web/gemfire/instances/ins_41415/locator41415state.dat

[MariaD]

Hi @411752 ,

I also tried cleaning the file and directory of the Cache Locator but nothing changes.

Regards, 

[MariaD]

Hi @StefanM 

thanks, I'll tried but the following error appear:

Am I doing something wrong? 

[StefanM]

It should work like this, independant of the other idea with gemfire/cache locator.

I would use the trace:
sh -xv ./tcruntime-ctl.sh status

[MariaD]

Hi @StefanM , 

Follows the results:

[sas@sas bin]$ sh -xv ./tcruntime-ctl.sh status
#!/bin/sh

# ---------------------------------------------------------------------------
# Copyright (c) 2010-2020 VMware, Inc. or its affiliates. All rights reserved.
# ---------------------------------------------------------------------------

##
# This version calls the top-level tcruntime-ctl.sh
# script
##

INSTALL_BASE="/sas/software/SASWebApplicationServer/9.4"
+ INSTALL_BASE=/sas/software/SASWebApplicationServer/9.4

#

os400=false
+ os400=false
darwin=false
+ darwin=false
case "`uname`" in
CYGWIN*) cygwin=true;;
OS400*) os400=true;;
Darwin*) darwin=true;;
esac
+ case "`uname`" in
uname
++ uname

# resolve links - $0 may be a softlink
PRG="$0"
+ PRG=./tcruntime-ctl.sh

while [ -h "$PRG" ] ; do
ls=`ls -ld "$PRG"`
link=`expr "$ls" : '.*-> \(.*\)$'`
if expr "$link" : '/.*' > /dev/null; then
PRG="$link"
else
PRG=`dirname "$PRG"`/"$link"
fi
done
+ '[' -h ./tcruntime-ctl.sh ']'

PRGDIR=`dirname "$PRG"`
dirname "$PRG"
++ dirname ./tcruntime-ctl.sh
+ PRGDIR=.
PRGDIR=`cd "$PRGDIR"; pwd -P`
cd "$PRGDIR"; pwd -P
++ cd .
++ pwd -P
+ PRGDIR=/sas/config/Lev1/Web/WebAppServer/SASServer1_1/bin

#Guess instance name
BASEDIR=`cd "$PRGDIR/../." ; pwd -P`
cd "$PRGDIR/../." ; pwd -P
++ cd /sas/config/Lev1/Web/WebAppServer/SASServer1_1/bin/../.
++ pwd -P
+ BASEDIR=/sas/config/Lev1/Web/WebAppServer/SASServer1_1
INSTANCE_NAME=`basename "$BASEDIR"`
basename "$BASEDIR"
++ basename /sas/config/Lev1/Web/WebAppServer/SASServer1_1
+ INSTANCE_NAME=SASServer1_1

#Absolute path
PRGDIR=`cd "$PRGDIR/../../." ; pwd -P`
cd "$PRGDIR/../../." ; pwd -P
++ cd /sas/config/Lev1/Web/WebAppServer/SASServer1_1/bin/../../.
++ pwd -P
+ PRGDIR=/sas/config/Lev1/Web/WebAppServer

#Set instance base if it's not already set
if [ -z "$INSTANCE_BASE" ]; then
INSTANCE_BASE="$PRGDIR"
export INSTANCE_BASE
fi
+ '[' -z '' ']'
+ INSTANCE_BASE=/sas/config/Lev1/Web/WebAppServer
+ export INSTANCE_BASE

EXECUTABLE=tcruntime-ctl.sh
+ EXECUTABLE=tcruntime-ctl.sh

PROGRAM=""
+ PROGRAM=

#first check install base

# Check that target executable exists
if [ -d "$INSTALL_BASE" ]; then
if [ -x "$INSTALL_BASE"/"$EXECUTABLE" ]; then
PROGRAM="$INSTALL_BASE"/"$EXECUTABLE"
elif $os400; then
# -x will Only work on the os400 if the files are:
# 1. owned by the user
# 2. owned by the PRIMARY group of the user
# this will not work if the user belongs in secondary groups
eval
PROGRAM="$INSTALL_BASE"/"$EXECUTABLE"
else
echo "ERROR Cannot find $INSTALL_BASE/$EXECUTABLE or it's not executable."
fi
fi
+ '[' -d /sas/software/SASWebApplicationServer/9.4 ']'
+ '[' -x /sas/software/SASWebApplicationServer/9.4/tcruntime-ctl.sh ']'
+ PROGRAM=/sas/software/SASWebApplicationServer/9.4/tcruntime-ctl.sh

if [ -z "$PROGRAM" ]; then
if [ ! -x "$PRGDIR"/"$EXECUTABLE" ]; then
echo "ERROR Cannot find $PRGDIR/$EXECUTABLE or it's not executable."
echo "ERROR One of these files is needed to run this program"
exit 1
else
PROGRAM="$PRGDIR"/"$EXECUTABLE"
fi
fi
+ '[' -z /sas/software/SASWebApplicationServer/9.4/tcruntime-ctl.sh ']'

if [ "verbose-status" = "$1" ]; then
echo "Derived instance name: $INSTANCE_NAME"
echo "Executing script: $PROGRAM"
fi
+ '[' verbose-status = status ']'
exec "$PROGRAM" "$INSTANCE_NAME" "$1" "$2"
+ exec /sas/software/SASWebApplicationServer/9.4/tcruntime-ctl.sh SASServer1_1 status ''
Invalid sytax.

Thanks,

[StefanM]

And this script (will not work even if all fine this way):

sh -xv /sas/software/SASWebApplicationServer/9.4/tcruntime-ctl.sh SASServer1_1 status ''

[MariaD]

Hi @StefanM ,

Nope, same thing: 

[sas@sas bin]$ sh -xv /sas/software/SASWebApplicationServer/9.4/tcruntime-ctl.sh SASServer1_1 status
#!/bin/sh

# ------------------------------------------------------------------------------
# Copyright (c) 2021 by SAS Institute Inc., Cary, NC USA 27513
# ------------------------------------------------------------------------------
#
# Invoke Tomcat's catalina shell script to control the web application server.
#
# ------------------------------------------------------------------------------

# Read the instance name
if [ "$#" -gt 0 ]; then
name=$1
shift
else
echo "Invalid sytax."
exit 1
fi
+ '[' 2 -gt 0 ']'
+ name=SASServer1_1
+ shift

# Read the command
if [ "$#" -gt 0 ]; then
command=$1
shift
else
echo "Invalid sytax."
exit 1
fi
+ '[' 1 -gt 0 ']'
+ command=status
+ shift

# Read the path to the instance
if [ "$#" -gt 1 ]; then
if [ "$1" = "-n" ]; then
# Shift off the recognized command switch
shift
# Remember and shift off the template name
instancedir=$1
shift
else
echo "Invalid syntax."
exit 1
fi
else
echo "Invalid sytax."
exit 1
fi
+ '[' 0 -gt 1 ']'
+ echo 'Invalid sytax.'
Invalid sytax.
+ exit 1