topic Re: Upgraded SAS 9.4 M3 to M7 but now services are not getting UP in Administration and Deployment

Upgraded SAS 9.4 M3 to M7 but now services are not getting UP

jayshree — Thu, 03 Jun 2021 22:03:31 GMT

There are 4 services not getting started .

Starting SAS Cache Locator Service: ins_41415

Error from abpve service , rest unable to find

OpenJDK 64-Bit Server VM warning: Using incremental CMS is deprecated and will likely be removed in a future release
Error: Could not find or load main class Dcom.sas.services.logging.disableRemoteList=true

SAS Web Server is NOT up
SAS Web App Server SASServer1_1 is NOT up
SAS Environment Manager is NOT up

Re: Upgraded SAS 9.4 M3 to M7 but now services are not getting UP

SASKiwi — Thu, 03 Jun 2021 22:13:52 GMT

Check the logs for those servers for any errors.

Try starting Environment Manager manually from the command line.

Re: Upgraded SAS 9.4 M3 to M7 but now services are not getting UP

jayshree — Thu, 03 Jun 2021 22:16:38 GMT

i am trying to start with sh sas.servers.mid but it is showing waiting and then came out mentioning below

waiting 120 ...
SAS Cache Locator Service ins_41415 is still in the process of starting up,
proceeding with additional servers.
Starting SAS Web Server
Starting SAS Web App Server: SASServer1_1
(this can take a while)
Starting SAS Environment Manager
Starting SAS Environment Manager Agent

Re: Upgraded SAS 9.4 M3 to M7 but now services are not getting UP

jayshree — Thu, 03 Jun 2021 22:30:27 GMT

Checked logs under each path :
/.../config/Lev1/Web/SASEnvironmentManager/server-5.8.0-EE/ --> NO Logs update

/../config/Lev1/Web/WebServer/logs --> --> NO Logs update

/../config/Lev1/Web/WebAppServer/SASServer1_1/logs
Error: Could not find or load main class Dcom.sas.services.logging.disableRemoteList=true

/ybs/saspatch/sas/config/Lev1/Web/gemfire/instances/ins_41415

Exception in processing request from XXXXX
com.gemstone.gemfire.SerializationException: Could not create an instance of com.gemstone.org.jgroups.stack.GossipData .
at com.gemstone.gemfire.internal.InternalDataSerializer.readDataSerializable(InternalDataSerializer.java:2419)
at com.gemstone.gemfire.internal.InternalDataSerializer.basicReadObject(InternalDataSerializer.java:2756)
at com.gemstone.gemfire.DataSerializer.readObject(DataSerializer.java:3217)
at com.gemstone.org.jgroups.stack.tcpserver.TcpServer$3.run(TcpServer.java:271)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.io.DataInputStream.readFully(DataInputStream.java:195)
at java.io.DataInputStream.readUTF(DataInputStream.java:609)
at java.io.DataInputStream.readUTF(DataInputStream.java:564)
at com.gemstone.org.jgroups.stack.GossipData.fromData(GossipData.java:139)
at com.gemstone.gemfire.internal.InternalDataSerializer.readDataSerializable(InternalDataSerializer.java:2407)

Service Status:

SAS Cache Locator Service ins_41415 is NOT up
SAS Web Server is NOT up
SAS Web App Server SASServer1_1 is NOT up
SAS Environment Manager is NOT up

Let me know what else i can validate to dig in ..

Re: Upgraded SAS 9.4 M3 to M7 but now services are not getting UP

SASKiwi — Thu, 03 Jun 2021 22:38:38 GMT

I suggest you raise a Tech Support track for this if you haven't done so already. Looks like your SAS runs on Unix and ours doesn't so I can't compare what you are seeing with ours.

Re: Upgraded SAS 9.4 M3 to M7 but now services are not getting UP

DJongman — Fri, 04 Jun 2021 09:13:17 GMT

When upgrading we normally delete next files, All logs and files wil be recreated after starting services.
(Also, if something is occurring, log-files are smaller to read)

stop all services.

Remove in directory: /srv/SASConfig/Lev1/Web/gemfire/instances/ins_41415 the next files
- gemfire.log
- locator41415state.dat
- locator41415views.log

/srv/SASConfig/Lev1/Web/activemq/data (remove all files, but keep subdirectories)

In all the directories: /srv/SASConfig/Lev/Web/Webappserver/SASServer*/logs remove files:

- catalina.out
- server.log
- gemfire.log

then restart all services

Re: Upgraded SAS 9.4 M3 to M7 but now services are not getting UP

jayshree — Fri, 04 Jun 2021 11:45:38 GMT

I followed above steps :
But no luck

/../config/Lev1/Web/WebAppServer/SASServer1_1/logs
cat catalina.out
OpenJDK 64-Bit Server VM warning: ignoring option PermSize=768m; support was removed in 8.0
OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=1536m; support was removed in 8.0
OpenJDK 64-Bit Server VM warning: Using incremental CMS is deprecated and will likely be removed in a future release
Error: Could not find or load main class Dcom.sas.services.logging.disableRemoteList=true

/..//config/Lev1/Web/activemq/data

ERROR | Could not accept connection : java.lang.InterruptedException | org.apache.activemq.broker.TransportConnector | ActiveMQ Transport Server Thread Handler:

/../config/Lev1/Web/gemfire/instances/ins_41415
gemfireCommand.log
[info 2021/06/04 12:39:16.610 BST <main> tid=0x1] Locator stopped
[severe 2021/06/04 12:39:16.611 BST <main> tid=0x1] Could not bind locator to null[41,415]

Re: Upgraded SAS 9.4 M3 to M7 but now services are not getting UP

Sajid01 — Sun, 06 Jun 2021 00:18:59 GMT

Hello @jayshree

The cache locator needs to be started before any of the Web /Web App/Ev Manager Server.
So address this first.

Follow the instructions here https://support.sas.com/kb/54/348.html .
Please let us know if the error still persists

Re: Upgraded SAS 9.4 M3 to M7 but now services are not getting UP

jayshree — Sun, 06 Jun 2021 16:01:50 GMT

Have deleted locatorNNNNNstate.dat file as per above suggested by DJongman , after that above error persist.

Re: Upgraded SAS 9.4 M3 to M7 but now services are not getting UP

Sajid01 — Sun, 06 Jun 2021 16:43:47 GMT

Hello @jayshree
No worries.
I have also faced this error in the past.
I believe catch locator is not starting because of this error. you shared in the last message.
[severe 2021/06/04 12:39:16.611 BST <main> tid=0x1] Could not bind locator to null[41,415]
Looks like an earlier instance of gemfire is running.
You can either find it and stop it manually or better to give a clean restart. I would prefer a clean restart.
By clean restart I mean stop SAS systematically, reboot the machine and start SAS.
As the locatorNNNNNstate.dat file is already deleted and if there is no other issue, the Server would start fine.
Please consider doing a clean restart.
Should you need further help, please do share the logs / console output.

Re: Upgraded SAS 9.4 M3 to M7 but now services are not getting UP

jayshree — Mon, 07 Jun 2021 15:05:15 GMT

Did fresh installation again today ..
All services are up except .. when checked in logs

SAS Web Server is NOT up --> No Logs/.... /config/Lev1/Web/WebServer/logs
SAS Environment Manager is NOT up -- > No Logs /.../config/Lev1/Web/SASEnvironmentManager/server-5.8.0-EE

Is there any other location to check logs

Re: Upgraded SAS 9.4 M3 to M7 but now services are not getting UP

Sajid01 — Mon, 07 Jun 2021 15:39:04 GMT

Please try manually starting those services.
First Webserver, then environment manager

Re: Upgraded SAS 9.4 M3 to M7 but now services are not getting UP

jayshree — Mon, 07 Jun 2021 16:14:28 GMT

config/Lev1/Web/WebServer/bin
sh httpdctl start
Starting pivotal httpd
httpd: Syntax error on line 185 of /.../config/Lev1/Web/WebServer/conf/httpd.conf: module bmx_module is built-in and can't be loaded
Server start FAILED

have not tried EM yet ..

Re: Upgraded SAS 9.4 M3 to M7 but now services are not getting UP

jayshree — Mon, 07 Jun 2021 17:09:10 GMT

all services started except

SAS Environment Manager is NOT up

/../ config/Lev1/Web/SASEnvironmentManager/server-5.8.0-EE/bin

Error :

sh hq-server.sh start
Java version was not found.

Any suggestion on this ...

Re: Upgraded SAS 9.4 M3 to M7 but now services are not getting UP

Sajid01 — Mon, 07 Jun 2021 17:52:38 GMT

Check for the directory [SASHOME]/SASPrivateJavaRunTimeEnvironment/9.4/jre.
This is the location for Java Run Time Environment.

The cause of the error is that hq-server.sh script considers that you do not have the proper version of Java.
I think you must create a ticket with SAS. They may help.

Re: Upgraded SAS 9.4 M3 to M7 but now services are not getting UP

jayshree — Mon, 07 Jun 2021 19:32:37 GMT

Checked the Deployment Summary :
Have updated Java as well .. but still getting the error ..

Installation
Completed successfully - SAS Private Java Runtime Environment (64-bit)

Any further help from forum would be appreciated

Re: Upgraded SAS 9.4 M3 to M7 but now services are not getting UP

Sajid01 — Mon, 07 Jun 2021 20:16:00 GMT

Create a file test.sh in the same directory as hq-server.sh.
For safety's sake do not make it executable
Then run it as sh test.sh.
Please post the output
The contents of the file are as follows.

###Replace [SASHOME] with actual path
T_JAVA_HOME=[SASHome]/SASPrivateJavaRunTimeEnvironment/9.4/jre
export ${T_JAVA_HOME}
${T_JAVA_HOME}/bin/java -version

Re: Upgraded SAS 9.4 M3 to M7 but now services are not getting UP

jayshree — Mon, 07 Jun 2021 20:42:14 GMT

sh test.sh
test.sh: line 2: export: `/.../SASPrivateJavaRuntimeEnvironment/9.4/jre': not a valid identifier
openjdk version "1.8.0_252"
OpenJDK Runtime Environment (Zulu 8.46.0.20-SA-linux64) (build 1.8.0_252-b14)
OpenJDK 64-Bit Server VM (Zulu 8.46.0.20-SA-linux64) (build 25.252-b14, mixed mode)

Re: Upgraded SAS 9.4 M3 to M7 but now services are not getting UP

Sajid01 — Tue, 08 Jun 2021 00:40:11 GMT

Please check if the path of SASPrivateJavaRuntimeEnvironment/9.4/jre in the same in hq-server.sh is correct or not.
I see you have the proper java installed,

Re: Upgraded SAS 9.4 M3 to M7 but now services are not getting UP

jayshree — Tue, 08 Jun 2021 03:47:59 GMT

I get correct path when i executed few below steps from hq-server.sh code

HQ_JAVA_HOME=/../sas/software/SASPrivateJavaRuntimeEnvironment/9.4/jre
export HQ_JAVA_HOME
JAVA_HOME=${HQ_JAVA_HOME}
HQ_JAVA="${JAVA_HOME}/bin/java"
echo $HQ_JAVA
/.../sas/software/SASPrivateJavaRuntimeEnvironment/9.4/jre/bin/java

hq-server.sh script

# -------------
# Begin HQ Server specific logic
# -------------
HQ_JAVA_HOME=/../sas/software/SASPrivateJavaRuntimeEnvironment/9.4/jre
export HQ_JAVA_HOME

if [ "x${HQ_JAVA_HOME}" != "x" ] ; then
JAVA_HOME=${HQ_JAVA_HOME}
elif [ -d ${SERVER_INSTALL_HOME}/jre ]; then
JAVA_HOME=${SERVER_INSTALL_HOME}/jre
elif [ "x$JAVA_HOME" = "x" ] ; then
case "`uname`" in
Darwin)
JAVA_HOME=/System/Library/Frameworks/JavaVM.framework/Home
;;
*)
echo "JAVA_HOME or HQ_JAVA_HOME must be set when invoking the server"
exit 1
;;
esac
fi

HQ_JAVA="${JAVA_HOME}/bin/java"
# verify that the java command actually exists
if [ ! -f "$HQ_JAVA" ]
then
echo Invalid Java Home detected at ${JAVA_HOME}
exit 1
fi

REQUIRED_VERSION_STRING=1.6
# Transform the required version string into a number that can be used in comparisons
REQUIRED_VERSION=`echo ${REQUIRED_VERSION_STRING} | sed -e 's;\.;0;g'`
# Check HQ_JAVA to see if Java version is adequate
if [ ${HQ_JAVA} ]
then
${HQ_JAVA} -version 2> /tmp/tmp.ver
VERSION=`cat /tmp/tmp.ver | grep "java version" | awk '{ print substr($3, 2, length($3)-2); }'`
rm /tmp/tmp.ver
VERSION=`echo $VERSION | awk '{ print substr($1, 1, 3); }' | sed -e 's;\.;0;g'`
if [ ${VERSION} ]
then
if [ ${VERSION} -lt ${REQUIRED_VERSION} ]
then
echo "Java version must be at least ${REQUIRED_VERSION_STRING}."
exit 1
fi
else
echo "Java version was not found."
exit 1
fi
fi