SASServ12_1 suddenly won't start up

[PriitL]

Hi!

This is my test environment. It was cloned, changed hostnames and passwords (including certificates and passphrases). Got almost everything (for example problems with DBRT but shouln't affect SASServer startup) up and running and suddenly after a while when I needed to restart the system I noticed that SASServer12_1 won't come up after restart. Didn't do anything. catalina.out says:

Could not load Logmanager "com.springsource.tcserver.serviceability.logging.TcServerLogManager"
java.lang.ClassNotFoundException: com.springsource.tcserver.serviceability.logging.TcServerLogManager
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.util.logging.LogManager$1.run(LogManager.java:195)
at java.util.logging.LogManager$1.run(LogManager.java:181)
at java.security.AccessController.doPrivileged(Native Method)
at java.util.logging.LogManager.<clinit>(LogManager.java:181)
at java.util.logging.Logger.demandLogger(Logger.java:448)
at java.util.logging.Logger.getLogger(Logger.java:502)
at org.apache.juli.logging.impl.Jdk14Logger.getLogger(Jdk14Logger.java:164)
at org.apache.juli.logging.impl.Jdk14Logger.<init>(Jdk14Logger.java:55)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.juli.logging.impl.LogFactoryImpl.createLogFromClass(LogFactoryImpl.java:1025)
at org.apache.juli.logging.impl.LogFactoryImpl.discoverLogImplementation(LogFactoryImpl.java:844)
at org.apache.juli.logging.impl.LogFactoryImpl.newInstance(LogFactoryImpl.java:541)
at org.apache.juli.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:292)
at org.apache.juli.logging.impl.LogFactoryImpl.getInstance(LogFactoryImpl.java:269)
at org.apache.juli.logging.LogFactory.getLog(LogFactory.java:657)
at org.apache.catalina.startup.Bootstrap.<clinit>(Bootstrap.java:52)
17:18:26,276 | WARN | [JmxSocketListener] | Detected LDAP configuration '/opt/sas/config/Lev1/Web/WebAppServer/SASServer12_1/conf/jaas.config', but there is no ldap configuration entry specified.
17:18:26,357 | INFO | [Catalina] | Initialization processed in 2059 ms
java.lang.RuntimeException
at com.atomikos.util.UniqueIdMgr.<init>(UniqueIdMgr.java:114)
at com.atomikos.util.UniqueIdMgr.<init>(UniqueIdMgr.java:90)
at com.atomikos.icatch.standalone.StandAloneTransactionManager.<init>(StandAloneTransactionManager.java:82)
at com.atomikos.icatch.standalone.UserTransactionServiceImp.createDefault(UserTransactionServiceImp.java:244)
at com.atomikos.icatch.standalone.UserTransactionServiceImp.init(UserTransactionServiceImp.java:265)
at com.atomikos.icatch.config.UserTransactionServiceImp.init(UserTransactionServiceImp.java:405)
at com.atomikos.icatch.config.UserTransactionServiceImp.init(UserTransactionServiceImp.java:569)
at com.atomikos.icatch.jta.UserTransactionManager.startupTransactionService(UserTransactionManager.java:89)
at com.atomikos.icatch.jta.UserTransactionManager.checkSetup(UserTransactionManager.java:77)
at com.atomikos.icatch.jta.UserTransactionManager.init(UserTransactionManager.java:142)
at com.sas.vfabrictcsvr.atomikos.AtomikosLifecycleListener.lifecycleEvent(AtomikosLifecycleListener.java:41)
at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:94)
at org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:395)
at org.apache.catalina.util.LifecycleBase.setState(LifecycleBase.java:339)
at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:786)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
at org.apache.catalina.startup.Catalina.start(Catalina.java:681)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:353)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:493)
17:18:26,905 | FATAL | [Catalina] | The required Server component failed to start so Tomcat is unable to start.
org.apache.catalina.LifecycleException: Failed to start component [StandardServer[-1]]
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:167)
at org.apache.catalina.startup.Catalina.start(Catalina.java:681)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:353)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:493)
Caused by: com.atomikos.icatch.SysException: Error in init(): null
at com.atomikos.icatch.standalone.UserTransactionServiceImp.init(UserTransactionServiceImp.java:333)
at com.atomikos.icatch.config.UserTransactionServiceImp.init(UserTransactionServiceImp.java:405)
at com.atomikos.icatch.config.UserTransactionServiceImp.init(UserTransactionServiceImp.java:569)
at com.atomikos.icatch.jta.UserTransactionManager.startupTransactionService(UserTransactionManager.java:89)
at com.atomikos.icatch.jta.UserTransactionManager.checkSetup(UserTransactionManager.java:77)
at com.atomikos.icatch.jta.UserTransactionManager.init(UserTransactionManager.java:142)
at com.sas.vfabrictcsvr.atomikos.AtomikosLifecycleListener.lifecycleEvent(AtomikosLifecycleListener.java:41)
at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:94)
at org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:395)
at org.apache.catalina.util.LifecycleBase.setState(LifecycleBase.java:339)
at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:786)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
... 7 more
Caused by: java.lang.RuntimeException
at com.atomikos.util.UniqueIdMgr.<init>(UniqueIdMgr.java:114)
at com.atomikos.util.UniqueIdMgr.<init>(UniqueIdMgr.java:90)
at com.atomikos.icatch.standalone.StandAloneTransactionManager.<init>(StandAloneTransactionManager.java:82)
at com.atomikos.icatch.standalone.UserTransactionServiceImp.createDefault(UserTransactionServiceImp.java:244)
at com.atomikos.icatch.standalone.UserTransactionServiceImp.init(UserTransactionServiceImp.java:265)
... 18 more
17:18:26,912 | WARN | [LifecycleBase] | Calling stop() on failed component [StandardServer[-1]] to trigger clean-up did not complete.
org.apache.catalina.LifecycleException: Failed to stop component [StandardServer[-1]]
at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:238)
at org.apache.catalina.util.LifecycleBase.destroy(LifecycleBase.java:264)
at org.apache.catalina.startup.Catalina.start(Catalina.java:685)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:353)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:493)
Caused by: org.apache.catalina.LifecycleException: An invalid Lifecycle transition was attempted ([before_stop]) for component [StandardService[Catalina]] in state [INITIALIZED]
at org.apache.catalina.util.LifecycleBase.invalidTransition(LifecycleBase.java:402)
at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:213)
at org.apache.catalina.core.StandardServer.stopInternal(StandardServer.java:814)
at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:226)
... 8 more

And that's it. The only line which should be suspicious is about "An invalid Lifecycle transition was attempted".

Is something expired? The licence is valid (currently until 31.jan) and all the others certificates also should be valid.

Also couldn't find any similar issue from community or from internet..

Any hint what should I look for?

Thanks!

PL

[JuanS_OCS]

Hello @PriitL ,

there are a few things that I would recommend:

1- have a proper entry in the hosts file, aligned with how your original server was created (FQDN, shortname, aliases, etc) and pointing them to the IP of the new server.

2- Ensure the new server is in similar subnet as the original, otherwise multicast addresses must be updated.

3- Stop all services and clean up the work/logs/temp files from each service (let me stress it, each service).

4- Start all services and validate the system

5- If it works, you should think about the change hostnames and rest of actions, in order to remove the temporary entry in the hosts file.

Still, this clone probably won't be ready to be a real production system, as this procedure is not officially supported by SAS Technical Support, and I can tell you that chances are high you might have challenges during the life cycle of this clone. If you use, you will assume all the risks.

Furthermore, this clone does need a new license you need to purchase at SAS Institute. Otherwise, legally speaking you would be incurring in one of the contract termination clauses defined in your supplement. Unless, of course, this clone is something discussed and agreed with your SAS Account Manager.

I hope it helps.

Best regards,

Juan

[PriitL · Accepted solution]

With the help of Support we figured out what happened.

But as a disclaimer- the following is at Your own risk. 🙂

First of all this can happen when the SASServer is started before previous instance is stopped completely. In other words, two copies of the WebAppServer are running at some moment. To fix:

1) Stop SAS services on mid-tier
2) be sure all the services are gone (kill -15 <process> if needed)
3) Clean/Archive the files in the folders:
../Lev1/Web/WebAppServer/SASServer12_1/logs/*
../Lev1/Web/WebAppServer/SASServer12_1/temp/*
../Lev1/Web/WebAppServer/SASServer12_1/work/*
../Lev1/Web/activemq/data/*
4) Start the SAS Services

The instructions were also to clear the activemq/data directory if these 4 steps don't work in the same logical way (stop, clear contents, start).

In my case this fixed the issue.

Further instructions said that if it still fails it might have something to do with the lib-directory (bunch of jar files).
They should be same files as the ones under for example SASServer1_1/lib.
If comparing the contents of the directories differ, maybe some of those jar files are corrupted. If so, then again stop all the services, backup, overwrite corrupted file(s) and try again.

Still at Your own risk. 🙂

PL

[SASKiwi]

@PriitL - Please mark your post as answered if you are OK with your answer.