Upgraded SAS 9.4 M3 to M7 but now services are not getting UP

[jayshree]

There are 4 services not getting started .

Starting SAS Cache Locator Service: ins_41415

Error from abpve service , rest unable to find 

OpenJDK 64-Bit Server VM warning: Using incremental CMS is deprecated and will likely be removed in a future release
Error: Could not find or load main class Dcom.sas.services.logging.disableRemoteList=true

SAS Web Server is NOT up
SAS Web App Server SASServer1_1 is NOT up
SAS Environment Manager is NOT up

[SASKiwi]

Check the logs for those servers for any errors.

Try starting Environment Manager manually from the command line.

[jayshree]

i am trying to start with sh sas.servers.mid but it is showing waiting and then came out mentioning below

waiting 120 ...
SAS Cache Locator Service ins_41415 is still in the process of starting up,
proceeding with additional servers.
Starting SAS Web Server
Starting SAS Web App Server: SASServer1_1
(this can take a while)
Starting SAS Environment Manager
Starting SAS Environment Manager Agent

[jayshree]

Checked logs under each path :
/.../config/Lev1/Web/SASEnvironmentManager/server-5.8.0-EE/ --> NO Logs update

/../config/Lev1/Web/WebServer/logs --> --> NO Logs update

/../config/Lev1/Web/WebAppServer/SASServer1_1/logs
Error: Could not find or load main class Dcom.sas.services.logging.disableRemoteList=true

/ybs/saspatch/sas/config/Lev1/Web/gemfire/instances/ins_41415

Exception in processing request from XXXXX
com.gemstone.gemfire.SerializationException: Could not create an instance of com.gemstone.org.jgroups.stack.GossipData .
at com.gemstone.gemfire.internal.InternalDataSerializer.readDataSerializable(InternalDataSerializer.java:2419)
at com.gemstone.gemfire.internal.InternalDataSerializer.basicReadObject(InternalDataSerializer.java:2756)
at com.gemstone.gemfire.DataSerializer.readObject(DataSerializer.java:3217)
at com.gemstone.org.jgroups.stack.tcpserver.TcpServer$3.run(TcpServer.java:271)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.io.DataInputStream.readFully(DataInputStream.java:195)
at java.io.DataInputStream.readUTF(DataInputStream.java:609)
at java.io.DataInputStream.readUTF(DataInputStream.java:564)
at com.gemstone.org.jgroups.stack.GossipData.fromData(GossipData.java:139)
at com.gemstone.gemfire.internal.InternalDataSerializer.readDataSerializable(InternalDataSerializer.java:2407)


Service Status:

SAS Cache Locator Service ins_41415 is NOT up
SAS Web Server is NOT up
SAS Web App Server SASServer1_1 is NOT up
SAS Environment Manager is NOT up

Let me know what else i can validate to dig in ..

[SASKiwi]

I suggest you raise a Tech Support track for this if you haven't done so already. Looks like your SAS runs on Unix and ours doesn't so I can't compare what you are seeing with ours.

[DJongman]

When upgrading we normally delete next files, All logs and files wil be recreated after starting services.
(Also, if something is occurring, log-files are smaller to read)

stop all services.

Remove in directory: /srv/SASConfig/Lev1/Web/gemfire/instances/ins_41415 the next files
- gemfire.log
- locator41415state.dat
- locator41415views.log

/srv/SASConfig/Lev1/Web/activemq/data (remove all files, but keep subdirectories)

In all the directories: /srv/SASConfig/Lev/Web/Webappserver/SASServer*/logs remove files:

- catalina.out
- server.log
- gemfire.log

then restart all services

[jayshree]

I followed above steps :
But no luck

/../config/Lev1/Web/WebAppServer/SASServer1_1/logs
cat catalina.out
OpenJDK 64-Bit Server VM warning: ignoring option PermSize=768m; support was removed in 8.0
OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=1536m; support was removed in 8.0
OpenJDK 64-Bit Server VM warning: Using incremental CMS is deprecated and will likely be removed in a future release
Error: Could not find or load main class Dcom.sas.services.logging.disableRemoteList=true


/..//config/Lev1/Web/activemq/data

ERROR | Could not accept connection : java.lang.InterruptedException | org.apache.activemq.broker.TransportConnector | ActiveMQ Transport Server Thread Handler:

/../config/Lev1/Web/gemfire/instances/ins_41415
gemfireCommand.log
[info 2021/06/04 12:39:16.610 BST <main> tid=0x1] Locator stopped
[severe 2021/06/04 12:39:16.611 BST <main> tid=0x1] Could not bind locator to null[41,415]

[Sajid01]

Hello @jayshree 

The cache locator needs to be started before any of the Web /Web App/Ev Manager Server.
So address this first.

Follow the instructions here https://support.sas.com/kb/54/348.html .
Please let us know if the error still persists

[jayshree]

Have deleted locatorNNNNNstate.dat file as per above suggested by DJongman , after that above error persist.

[Sajid01]

Hello @jayshree 
No worries. 
I have also faced this error in the past.
I believe catch locator is not starting because of this error. you shared in the last message.
[severe 2021/06/04 12:39:16.611 BST <main> tid=0x1] Could not bind locator to null[41,415]
Looks like an earlier instance of gemfire is running.
You can either find it and stop it manually or better to give a clean restart. I would prefer  a clean restart.
By clean restart I mean  stop SAS systematically, reboot the machine and start SAS.
As  the locatorNNNNNstate.dat file is already deleted  and if there is no other issue, the Server would start fine.
Please consider doing a clean restart.
Should you need further help, please do share the logs / console output.

[jayshree]

Did fresh installation again today ..
All services are up except .. when checked in logs

SAS Web Server is NOT up --> No Logs/.... /config/Lev1/Web/WebServer/logs
SAS Environment Manager is NOT up -- > No Logs /.../config/Lev1/Web/SASEnvironmentManager/server-5.8.0-EE

Is there any other location to check logs

[Sajid01]

Please try manually starting those services.
First Webserver, then environment manager

[jayshree]

config/Lev1/Web/WebServer/bin
sh httpdctl start
Starting pivotal httpd
httpd: Syntax error on line 185 of /.../config/Lev1/Web/WebServer/conf/httpd.conf: module bmx_module is built-in and can't be loaded
Server start FAILED

have not tried EM yet ..

[jayshree]

all services started except

SAS Environment Manager is NOT up

/../ config/Lev1/Web/SASEnvironmentManager/server-5.8.0-EE/bin

Error :

sh hq-server.sh start
Java version was not found.

Any suggestion on this ...

[Sajid01]

Check for the directory [SASHOME]/SASPrivateJavaRunTimeEnvironment/9.4/jre.
This is the location for Java Run Time Environment.

The cause of the error is that hq-server.sh script   considers that you do not have the proper version of Java.
I think you must create a ticket with SAS. They may help.

.