topic Re: Visual Analytics Web Applications Service Temporarily Unavailable in Administration and Deployment

Visual Analytics Web Applications Service Temporarily Unavailable

MarkPeskir — Wed, 06 Jul 2016 15:55:15 GMT

We have been running SAS VA 7.2 non-distributed on a Windows 2012 R2 server for several months now. Suddenly, all of the web applications (VA, Web Administration, Environment Manager, SAS Studio) running on this box now get "Service Temporarily Unavailable". Rebooting the box grants about 10 minutes of uptime, then it goes down again.

The windows server itself shows all expected SAS services running, and Management Console connects just fine and shows all configuration settings. The issue seems to be focused to just the web server (or web app server).

I have a support ticket submitted, but now we've been down for days and I'm recruiting all the help I can get.

Re: Visual Analytics Web Applications Service Temporarily Unavailable

dpage — Wed, 06 Jul 2016 15:59:29 GMT

What's in the LevN/Web/WebAppServer/SASServer12_1/logs/server.log (possibly SASServer1 or SASServer if you're single managed server)? If you're seeing that error message it means the WebServer is okay, but the web app server is not - the reverse proxy can't communicate with the backend server - typically that means additional information is available either in a particular application log in LevN/Web/Logs/SASServerN/ or the aforementioned server.log. Likely culprit is the JVM exiting for some reason or thrashing in garbage collection to the point it becomes unresponsive.

Re: Visual Analytics Web Applications Service Temporarily Unavailable

MarkPeskir — Wed, 06 Jul 2016 16:20:08 GMT

Most recent error message, written shortly after a reboot: (internal hostname obfuscated by me)

2016-07-06 11:19:45,694 ERROR (localhost-startStop-6) [org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/SASDeploymentBackup]] Exception sending context initialized event to listener instance of class org.springframework.web.context.ContextLoaderListener
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'identityCache': Invocation of init method failed; nested exception is com.gemstone.gemfire.distributed.DistributedSystemDisconnectedException: GemFire on [[INTERNAL HOSTNAME]]<v482>:30329/59464 started at Wed Jul 06 11:11:20 EDT 2016: Message distribution has terminated

[[followed by java stack trace]]

Re: Visual Analytics Web Applications Service Temporarily Unavailable

MarkPeskir — Wed, 06 Jul 2016 16:44:52 GMT

Also, SAS support helped us enable GC logging, and I see this behavior [abridged for clarity]:

{Heap before GC invocations=34 (full 0):
par new generation total 419392K, used 404067K
eden space 372800K, 100% used
from space 46592K, 67% used
to space 46592K, 0% used
2016-07-06T12:36:51 [Times: user=0.05 sys=0.02, real=0.02 secs]
Heap after GC invocations=35 (full 0):
par new generation total 419392K, used 26187K
eden space 372800K, 0% used
from space 46592K, 56% used
to space 46592K, 0% used

372 of our 512 GB memory goes from 0 to 100% in two seconds? This goes back and forth over and over several times, in very short intervals. I'm not fluent in log files, but this seems not good.

Re: Visual Analytics Web Applications Service Temporarily Unavailable

dpage — Wed, 06 Jul 2016 17:19:52 GMT

Sounds like support has you on the right track, none of that seems especially bad though (your eden space isn't the entire heap, see

http://www.oracle.com/technetwork/tutorials/tutorials-1876574.html#t2 for a decent summary)