BookmarkSubscribeRSS Feed
bheinsius
Lapis Lazuli | Level 10

Hi,

 

After returning from vacation my midtier is not working anymore.

 

When starting SASServer1_1 all webapps deploy fine except for SASWIPServices.

 

WIPServices9.4.log listed below.

 

Any ideas?

 

Thanks,

Bart

 

 

 

2018-09-12 00:09:14,947 [localhost-startStop-3] INFO [unknown] com.sas.svcs.springframework.config.EnvConfigurerBeanDefinitionParser - Using pre-existing environment properties bean named 'environmentProperties'

2018-09-12 00:10:26,359 [localhost-startStop-3] WARN [unknown] com.sas.services.session.SessionContext - Forced destruction of the session context by the Session Service. Session Context=5e857f2a1118a054:-6b3a7250:165caaa1348:-3011

2018-09-12 00:10:26,367 [localhost-startStop-3] WARN [unknown] com.sas.services.session.SessionContext - Forced destruction of the session context by the Session Service. Session Context=5e857f2a1118a054:-6b3a7250:165caaa1348:-301a

2018-09-12 00:10:26,378 [localhost-startStop-3] ERROR [unknown] org.springframework.web.context.ContextLoader - Context initialization failed

org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'rootFolderNavigationServiceImpl': Injection of autowired dependencies failed; nested exception is com.gemstone.gemfire.SerializationException: Class not found

at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:298)

at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1148)

at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:517)

at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)

at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:293)

at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:223)

at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:290)

at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:191)

at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:638)

at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:942)

at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:482)

at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:410)

at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:306)

at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:112)

at com.sas.svcs.webapp.servlet.springframework.DefaultContextLoaderListener.contextInitialized(DefaultContextLoaderListener.java:44)

at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4745)

at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5207)

at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)

at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:752)

at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:728)

at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:734)

at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:596)

at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1805)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:473)

at java.util.concurrent.FutureTask.run(FutureTask.java:262)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1152)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:622)

at java.lang.Thread.run(Thread.java:748)

Caused by: com.gemstone.gemfire.SerializationException: Class not found

at com.gemstone.gemfire.distributed.internal.ReplyException.handleAsUnexpected(ReplyException.java:84)

at com.gemstone.gemfire.internal.cache.DistributedCacheOperation.waitForAckIfNeeded(DistributedCacheOperation.java:584)

at com.gemstone.gemfire.internal.cache.DistributedCacheOperation.distribute(DistributedCacheOperation.java:501)

at com.gemstone.gemfire.internal.cache.DistributedRegion.distributeUpdate(DistributedRegion.java:363)

at com.gemstone.gemfire.internal.cache.DistributedRegion.basicPutPart3(DistributedRegion.java:352)

at com.gemstone.gemfire.internal.cache.AbstractRegionMap.basicPut(AbstractRegionMap.java:2300)

at com.gemstone.gemfire.internal.cache.LocalRegion.virtualPut(LocalRegion.java:5085)

at com.gemstone.gemfire.internal.cache.DistributedRegion.virtualPut(DistributedRegion.java:285)

at com.gemstone.gemfire.internal.cache.LocalRegionDataView.putEntry(LocalRegionDataView.java:100)

at com.gemstone.gemfire.internal.cache.LocalRegion.basicPut(LocalRegion.java:4577)

at com.gemstone.gemfire.internal.cache.LocalRegion.validatedPut(LocalRegion.java:1610)

at com.gemstone.gemfire.internal.cache.LocalRegion.put(LocalRegion.java:1592)

at com.gemstone.gemfire.internal.cache.AbstractRegion.put(AbstractRegion.java:232)

at org.springframework.data.gemfire.support.GemfireCache.put(GemfireCache.java:68)

at com.sas.svcs.data.caches.gemfire.springframework.DistributedEvictionCache.put(DistributedEvictionCache.java:151)

at org.springframework.cache.interceptor.CacheAspectSupport.update(CacheAspectSupport.java:351)

at org.springframework.cache.interceptor.CacheAspectSupport.execute(CacheAspectSupport.java:214)

at org.springframework.cache.interceptor.CacheInterceptor.invoke(CacheInterceptor.java:66)

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)

at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)

at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)

at com.sun.proxy.$Proxy2242.getAllTypes(Unknown Source)

at com.sas.svcs.content.folder.impl.RootFolderNavigationServiceImpl.setObjectTypeService(RootFolderNavigationServiceImpl.java:54)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:606)

at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredMethodElement.inject(AutowiredAnnotationBeanPostProcessor.java:611)

at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:88)

at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:295)

... 27 more

Caused by: java.lang.ClassNotFoundException: com.sas.svcs.data.caches.gemfire.springframework.DistributedEvictionCache$KeyWrapper

at java.net.URLClassLoader$1.run(URLClassLoader.java:359)

at java.net.URLClassLoader$1.run(URLClassLoader.java:348)

at java.security.AccessController.doPrivileged(Native Method)

at java.net.URLClassLoader.findClass(URLClassLoader.java:347)

at java.lang.ClassLoader.loadClass(ClassLoader.java:425)

at java.lang.ClassLoader.loadClass(ClassLoader.java:358)

at java.lang.Class.forName0(Native Method)

at java.lang.Class.forName(Class.java:278)

at java.io.ObjectInputStream.resolveClass(ObjectInputStream.java:682)

at com.gemstone.gemfire.internal.InternalDataSerializer$DSObjectInputStream.resolveClass(InternalDataSerializer.java:3358)

at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1862)

at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1745)

at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2034)

at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1567)

at java.io.ObjectInputStream.readObject(ObjectInputStream.java:427)

at com.gemstone.gemfire.internal.InternalDataSerializer.basicReadObject(InternalDataSerializer.java:2790)

at com.gemstone.gemfire.DataSerializer.readObject(DataSerializer.java:3217)

at com.gemstone.gemfire.internal.cache.UpdateOperation$UpdateMessage.fromData(UpdateOperation.java:317)

at com.gemstone.gemfire.internal.DSFIDFactory.readUpdateMessage(DSFIDFactory.java:981)

at com.gemstone.gemfire.internal.DSFIDFactory.create(DSFIDFactory.java:205)

at com.gemstone.gemfire.internal.InternalDataSerializer.readDSFID(InternalDataSerializer.java:2524)

at com.gemstone.gemfire.internal.tcp.Connection.processNIOBuffer(Connection.java:3463)

at com.gemstone.gemfire.internal.tcp.Connection.runNioReader(Connection.java:1773)

at com.gemstone.gemfire.internal.tcp.Connection.run(Connection.java:1654)

... 1 more

2018-09-12 00:10:26,480 [localhost-startStop-3] ERROR [unknown] org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/SASWIPServices] - Exception sending context initialized event to listener instance of class com.sas.svcs.webapp.servlet.springframework.DefaultContextLoaderListener

org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'rootFolderNavigationServiceImpl': Injection of autowired dependencies failed; nested exception is com.gemstone.gemfire.SerializationException: Class not found

at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:298)

at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1148)

at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:517)

at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)

at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:293)

at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:223)

at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:290)

at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:191)

at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:638)

at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:942)

at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:482)

at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:410)

at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:306)

at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:112)

at com.sas.svcs.webapp.servlet.springframework.DefaultContextLoaderListener.contextInitialized(DefaultContextLoaderListener.java:44)

at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4745)

at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5207)

at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)

at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:752)

at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:728)

at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:734)

at org.apache.catalina.startup.HostConfig.deployDescriptor(HostConfig.java:596)

at org.apache.catalina.startup.HostConfig$DeployDescriptor.run(HostConfig.java:1805)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:473)

at java.util.concurrent.FutureTask.run(FutureTask.java:262)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1152)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:622)

at java.lang.Thread.run(Thread.java:748)

Caused by: com.gemstone.gemfire.SerializationException: Class not found

at com.gemstone.gemfire.distributed.internal.ReplyException.handleAsUnexpected(ReplyException.java:84)

at com.gemstone.gemfire.internal.cache.DistributedCacheOperation.waitForAckIfNeeded(DistributedCacheOperation.java:584)

at com.gemstone.gemfire.internal.cache.DistributedCacheOperation.distribute(DistributedCacheOperation.java:501)

at com.gemstone.gemfire.internal.cache.DistributedRegion.distributeUpdate(DistributedRegion.java:363)

at com.gemstone.gemfire.internal.cache.DistributedRegion.basicPutPart3(DistributedRegion.java:352)

at com.gemstone.gemfire.internal.cache.AbstractRegionMap.basicPut(AbstractRegionMap.java:2300)

at com.gemstone.gemfire.internal.cache.LocalRegion.virtualPut(LocalRegion.java:5085)

at com.gemstone.gemfire.internal.cache.DistributedRegion.virtualPut(DistributedRegion.java:285)

at com.gemstone.gemfire.internal.cache.LocalRegionDataView.putEntry(LocalRegionDataView.java:100)

at com.gemstone.gemfire.internal.cache.LocalRegion.basicPut(LocalRegion.java:4577)

at com.gemstone.gemfire.internal.cache.LocalRegion.validatedPut(LocalRegion.java:1610)

at com.gemstone.gemfire.internal.cache.LocalRegion.put(LocalRegion.java:1592)

at com.gemstone.gemfire.internal.cache.AbstractRegion.put(AbstractRegion.java:232)

at org.springframework.data.gemfire.support.GemfireCache.put(GemfireCache.java:68)

at com.sas.svcs.data.caches.gemfire.springframework.DistributedEvictionCache.put(DistributedEvictionCache.java:151)

at org.springframework.cache.interceptor.CacheAspectSupport.update(CacheAspectSupport.java:351)

at org.springframework.cache.interceptor.CacheAspectSupport.execute(CacheAspectSupport.java:214)

at org.springframework.cache.interceptor.CacheInterceptor.invoke(CacheInterceptor.java:66)

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)

at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)

at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)

at com.sun.proxy.$Proxy2242.getAllTypes(Unknown Source)

at com.sas.svcs.content.folder.impl.RootFolderNavigationServiceImpl.setObjectTypeService(RootFolderNavigationServiceImpl.java:54)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:606)

at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredMethodElement.inject(AutowiredAnnotationBeanPostProcessor.java:611)

at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:88)

at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:295)

... 27 more

Caused by: java.lang.ClassNotFoundException: com.sas.svcs.data.caches.gemfire.springframework.DistributedEvictionCache$KeyWrapper

at java.net.URLClassLoader$1.run(URLClassLoader.java:359)

at java.net.URLClassLoader$1.run(URLClassLoader.java:348)

at java.security.AccessController.doPrivileged(Native Method)

at java.net.URLClassLoader.findClass(URLClassLoader.java:347)

at java.lang.ClassLoader.loadClass(ClassLoader.java:425)

at java.lang.ClassLoader.loadClass(ClassLoader.java:358)

at java.lang.Class.forName0(Native Method)

at java.lang.Class.forName(Class.java:278)

at java.io.ObjectInputStream.resolveClass(ObjectInputStream.java:682)

at com.gemstone.gemfire.internal.InternalDataSerializer$DSObjectInputStream.resolveClass(InternalDataSerializer.java:3358)

at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1862)

at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1745)

at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2034)

at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1567)

at java.io.ObjectInputStream.readObject(ObjectInputStream.java:427)

at com.gemstone.gemfire.internal.InternalDataSerializer.basicReadObject(InternalDataSerializer.java:2790)

at com.gemstone.gemfire.DataSerializer.readObject(DataSerializer.java:3217)

at com.gemstone.gemfire.internal.cache.UpdateOperation$UpdateMessage.fromData(UpdateOperation.java:317)

at com.gemstone.gemfire.internal.DSFIDFactory.readUpdateMessage(DSFIDFactory.java:981)

at com.gemstone.gemfire.internal.DSFIDFactory.create(DSFIDFactory.java:205)

at com.gemstone.gemfire.internal.InternalDataSerializer.readDSFID(InternalDataSerializer.java:2524)

at com.gemstone.gemfire.internal.tcp.Connection.processNIOBuffer(Connection.java:3463)

at com.gemstone.gemfire.internal.tcp.Connection.runNioReader(Connection.java:1773)

at com.gemstone.gemfire.internal.tcp.Connection.run(Connection.java:1654)

... 1 more

 

3 REPLIES 3
JuanS_OCS
Amethyst | Level 16

Hello @bheinsius

 

if no one made changes to the environment, I would recommend to do some basic maintenance as first approach:

 

- stop middle tier completely

- clean / archive logs from SASServerN_M, JMS and Cache Locator

- delete contents of /tmp and /work folders of SASServerN_M and contents of JMS (data folder) and the Cache Locator (except the config file)

- Restart services, if possible, manually and checking the logs with a tail tool, to see if problems starts somewhere else.

 

 

Apart from this: do you have the VA Auditing enabled or the EMI Framework? Sometimes, this might happen when a few tables from WIP database grow "too" big, because SASServer1_1 loads in its JVM memory part of this data. You could check this also by checking, on SASServer1_1 start-time, its Heap Size.

 

Please let us know how it goes.

 

Kind regards,

 

Juan

bheinsius
Lapis Lazuli | Level 10

Hi @JuanS_OCS

 

Thanks for your suggestions.

I had tried/investigated all you suggest but it did not help.

 

As a last resort I rebooted all tiers (meta/compute/web) and that solved the problem.

The problem may have been caused by an incorrect startup order of the tiers after scheduled maintenance (Windows), as we have not set that up yet.

 

Regards,

Bart

JuanS_OCS
Amethyst | Level 16

Hey @bheinsius,

 

great, glad to know you solved your own problem 🙂

suga badge.PNGThe SAS Users Group for Administrators (SUGA) is open to all SAS administrators and architects who install, update, manage or maintain a SAS deployment. 

Join SUGA 

Get Started with SAS Information Catalog in SAS Viya

SAS technical trainer Erin Winters shows you how to explore assets, create new data discovery agents, schedule data discovery agents, and much more.

Find more tutorials on the SAS Users YouTube channel.

Discussion stats
  • 3 replies
  • 3435 views
  • 1 like
  • 2 in conversation