Hi,
After returning from vacation my midtier is not working anymore.
When starting SASServer1_1 all webapps deploy fine except for SASWIPServices.
WIPServices9.4.log listed below.
Any ideas?
Thanks,
Bart
2018-09-12 00:09:14,947 [localhost-startStop-3] INFO [unknown] com.sas.svcs.springframework.config.EnvConfigurerBeanDefinitionParser - Using pre-existing environment properties bean named 'environmentProperties'
2018-09-12 00:10:26,359 [localhost-startStop-3] WARN [unknown] com.sas.services.session.SessionContext - Forced destruction of the session context by the Session Service. Session Context=5e857f2a1118a054:-6b3a7250:165caaa1348:-3011
2018-09-12 00:10:26,367 [localhost-startStop-3] WARN [unknown] com.sas.services.session.SessionContext - Forced destruction of the session context by the Session Service. Session Context=5e857f2a1118a054:-6b3a7250:165caaa1348:-301a
2018-09-12 00:10:26,378 [localhost-startStop-3] ERROR [unknown] org.springframework.web.context.ContextLoader - Context initialization failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'rootFolderNavigationServiceImpl': Injection of autowired dependencies failed; nested exception is com.gemstone.gemfire.SerializationException: Class not found
at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:298)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1148)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:517)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:293)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:223)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:290)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:191)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:638)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:942)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:482)
at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:410)
at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:306)
at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:112)
at com.sas.svcs.webapp.servlet.springframework.DefaultContextLoaderListener.contextInitialized(DefaultContextLoaderListener.java:44)
at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4745)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5207)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:752)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:728)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:734)
at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:596)
at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1805)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:473)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1152)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:622)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.gemstone.gemfire.SerializationException: Class not found
at com.gemstone.gemfire.distributed.internal.ReplyException.handleAsUnexpected(ReplyException.java:84)
at com.gemstone.gemfire.internal.cache.DistributedCacheOperation.waitForAckIfNeeded(DistributedCacheOperation.java:584)
at com.gemstone.gemfire.internal.cache.DistributedCacheOperation.distribute(DistributedCacheOperation.java:501)
at com.gemstone.gemfire.internal.cache.DistributedRegion.distributeUpdate(DistributedRegion.java:363)
at com.gemstone.gemfire.internal.cache.DistributedRegion.basicPutPart3(DistributedRegion.java:352)
at com.gemstone.gemfire.internal.cache.AbstractRegionMap.basicPut(AbstractRegionMap.java:2300)
at com.gemstone.gemfire.internal.cache.LocalRegion.virtualPut(LocalRegion.java:5085)
at com.gemstone.gemfire.internal.cache.DistributedRegion.virtualPut(DistributedRegion.java:285)
at com.gemstone.gemfire.internal.cache.LocalRegionDataView.putEntry(LocalRegionDataView.java:100)
at com.gemstone.gemfire.internal.cache.LocalRegion.basicPut(LocalRegion.java:4577)
at com.gemstone.gemfire.internal.cache.LocalRegion.validatedPut(LocalRegion.java:1610)
at com.gemstone.gemfire.internal.cache.LocalRegion.put(LocalRegion.java:1592)
at com.gemstone.gemfire.internal.cache.AbstractRegion.put(AbstractRegion.java:232)
at org.springframework.data.gemfire.support.GemfireCache.put(GemfireCache.java:68)
at com.sas.svcs.data.caches.gemfire.springframework.DistributedEvictionCache.put(DistributedEvictionCache.java:151)
at org.springframework.cache.interceptor.CacheAspectSupport.update(CacheAspectSupport.java:351)
at org.springframework.cache.interceptor.CacheAspectSupport.execute(CacheAspectSupport.java:214)
at org.springframework.cache.interceptor.CacheInterceptor.invoke(CacheInterceptor.java:66)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at com.sun.proxy.$Proxy2242.getAllTypes(Unknown Source)
at com.sas.svcs.content.folder.impl.RootFolderNavigationServiceImpl.setObjectTypeService(RootFolderNavigationServiceImpl.java:54)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredMethodElement.inject(AutowiredAnnotationBeanPostProcessor.java:611)
at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:88)
at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:295)
... 27 more
Caused by: java.lang.ClassNotFoundException: com.sas.svcs.data.caches.gemfire.springframework.DistributedEvictionCache$KeyWrapper
at java.net.URLClassLoader$1.run(URLClassLoader.java:359)
at java.net.URLClassLoader$1.run(URLClassLoader.java:348)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:347)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:278)
at java.io.ObjectInputStream.resolveClass(ObjectInputStream.java:682)
at com.gemstone.gemfire.internal.InternalDataSerializer$DSObjectInputStream.resolveClass(InternalDataSerializer.java:3358)
at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1862)
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1745)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2034)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1567)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:427)
at com.gemstone.gemfire.internal.InternalDataSerializer.basicReadObject(InternalDataSerializer.java:2790)
at com.gemstone.gemfire.DataSerializer.readObject(DataSerializer.java:3217)
at com.gemstone.gemfire.internal.cache.UpdateOperation$UpdateMessage.fromData(UpdateOperation.java:317)
at com.gemstone.gemfire.internal.DSFIDFactory.readUpdateMessage(DSFIDFactory.java:981)
at com.gemstone.gemfire.internal.DSFIDFactory.create(DSFIDFactory.java:205)
at com.gemstone.gemfire.internal.InternalDataSerializer.readDSFID(InternalDataSerializer.java:2524)
at com.gemstone.gemfire.internal.tcp.Connection.processNIOBuffer(Connection.java:3463)
at com.gemstone.gemfire.internal.tcp.Connection.runNioReader(Connection.java:1773)
at com.gemstone.gemfire.internal.tcp.Connection.run(Connection.java:1654)
... 1 more
2018-09-12 00:10:26,480 [localhost-startStop-3] ERROR [unknown] org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/SASWIPServices] - Exception sending context initialized event to listener instance of class com.sas.svcs.webapp.servlet.springframework.DefaultContextLoaderListener
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'rootFolderNavigationServiceImpl': Injection of autowired dependencies failed; nested exception is com.gemstone.gemfire.SerializationException: Class not found
at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:298)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1148)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:517)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:293)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:223)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:290)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:191)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:638)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:942)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:482)
at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:410)
at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:306)
at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:112)
at com.sas.svcs.webapp.servlet.springframework.DefaultContextLoaderListener.contextInitialized(DefaultContextLoaderListener.java:44)
at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4745)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5207)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:752)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:728)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:734)
at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:596)
at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1805)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:473)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1152)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:622)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.gemstone.gemfire.SerializationException: Class not found
at com.gemstone.gemfire.distributed.internal.ReplyException.handleAsUnexpected(ReplyException.java:84)
at com.gemstone.gemfire.internal.cache.DistributedCacheOperation.waitForAckIfNeeded(DistributedCacheOperation.java:584)
at com.gemstone.gemfire.internal.cache.DistributedCacheOperation.distribute(DistributedCacheOperation.java:501)
at com.gemstone.gemfire.internal.cache.DistributedRegion.distributeUpdate(DistributedRegion.java:363)
at com.gemstone.gemfire.internal.cache.DistributedRegion.basicPutPart3(DistributedRegion.java:352)
at com.gemstone.gemfire.internal.cache.AbstractRegionMap.basicPut(AbstractRegionMap.java:2300)
at com.gemstone.gemfire.internal.cache.LocalRegion.virtualPut(LocalRegion.java:5085)
at com.gemstone.gemfire.internal.cache.DistributedRegion.virtualPut(DistributedRegion.java:285)
at com.gemstone.gemfire.internal.cache.LocalRegionDataView.putEntry(LocalRegionDataView.java:100)
at com.gemstone.gemfire.internal.cache.LocalRegion.basicPut(LocalRegion.java:4577)
at com.gemstone.gemfire.internal.cache.LocalRegion.validatedPut(LocalRegion.java:1610)
at com.gemstone.gemfire.internal.cache.LocalRegion.put(LocalRegion.java:1592)
at com.gemstone.gemfire.internal.cache.AbstractRegion.put(AbstractRegion.java:232)
at org.springframework.data.gemfire.support.GemfireCache.put(GemfireCache.java:68)
at com.sas.svcs.data.caches.gemfire.springframework.DistributedEvictionCache.put(DistributedEvictionCache.java:151)
at org.springframework.cache.interceptor.CacheAspectSupport.update(CacheAspectSupport.java:351)
at org.springframework.cache.interceptor.CacheAspectSupport.execute(CacheAspectSupport.java:214)
at org.springframework.cache.interceptor.CacheInterceptor.invoke(CacheInterceptor.java:66)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at com.sun.proxy.$Proxy2242.getAllTypes(Unknown Source)
at com.sas.svcs.content.folder.impl.RootFolderNavigationServiceImpl.setObjectTypeService(RootFolderNavigationServiceImpl.java:54)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredMethodElement.inject(AutowiredAnnotationBeanPostProcessor.java:611)
at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:88)
at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:295)
... 27 more
Caused by: java.lang.ClassNotFoundException: com.sas.svcs.data.caches.gemfire.springframework.DistributedEvictionCache$KeyWrapper
at java.net.URLClassLoader$1.run(URLClassLoader.java:359)
at java.net.URLClassLoader$1.run(URLClassLoader.java:348)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:347)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:278)
at java.io.ObjectInputStream.resolveClass(ObjectInputStream.java:682)
at com.gemstone.gemfire.internal.InternalDataSerializer$DSObjectInputStream.resolveClass(InternalDataSerializer.java:3358)
at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1862)
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1745)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2034)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1567)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:427)
at com.gemstone.gemfire.internal.InternalDataSerializer.basicReadObject(InternalDataSerializer.java:2790)
at com.gemstone.gemfire.DataSerializer.readObject(DataSerializer.java:3217)
at com.gemstone.gemfire.internal.cache.UpdateOperation$UpdateMessage.fromData(UpdateOperation.java:317)
at com.gemstone.gemfire.internal.DSFIDFactory.readUpdateMessage(DSFIDFactory.java:981)
at com.gemstone.gemfire.internal.DSFIDFactory.create(DSFIDFactory.java:205)
at com.gemstone.gemfire.internal.InternalDataSerializer.readDSFID(InternalDataSerializer.java:2524)
at com.gemstone.gemfire.internal.tcp.Connection.processNIOBuffer(Connection.java:3463)
at com.gemstone.gemfire.internal.tcp.Connection.runNioReader(Connection.java:1773)
at com.gemstone.gemfire.internal.tcp.Connection.run(Connection.java:1654)
... 1 more
Hello @bheinsius,
if no one made changes to the environment, I would recommend to do some basic maintenance as first approach:
- stop middle tier completely
- clean / archive logs from SASServerN_M, JMS and Cache Locator
- delete contents of /tmp and /work folders of SASServerN_M and contents of JMS (data folder) and the Cache Locator (except the config file)
- Restart services, if possible, manually and checking the logs with a tail tool, to see if problems starts somewhere else.
Apart from this: do you have the VA Auditing enabled or the EMI Framework? Sometimes, this might happen when a few tables from WIP database grow "too" big, because SASServer1_1 loads in its JVM memory part of this data. You could check this also by checking, on SASServer1_1 start-time, its Heap Size.
Please let us know how it goes.
Kind regards,
Juan
Hi @JuanS_OCS
Thanks for your suggestions.
I had tried/investigated all you suggest but it did not help.
As a last resort I rebooted all tiers (meta/compute/web) and that solved the problem.
The problem may have been caused by an incorrect startup order of the tiers after scheduled maintenance (Windows), as we have not set that up yet.
Regards,
Bart
The SAS Users Group for Administrators (SUGA) is open to all SAS administrators and architects who install, update, manage or maintain a SAS deployment.
SAS technical trainer Erin Winters shows you how to explore assets, create new data discovery agents, schedule data discovery agents, and much more.
Find more tutorials on the SAS Users YouTube channel.