Exploring, predicting and reporting with SAS Visual Analytics and SAS Visual Statistics

SAS VA Node is down

Reply
Contributor
Posts: 72

SAS VA Node is down

Hello,

Our SAS VA head node is going down on a weekly basis and we need to re-start it again. We are not getting any clue from the log and wondering what could be the possibilities / reasons and what we need to do to debug this issue so this won't happen again.

Thanks in advance !

SAS Super FREQ
Posts: 556

Re: SAS VA Node is down

@SAS_1001,

 

Your LASR server is running in distributed or non-distributed mode? Do you have any security scanners in your network?

Contributor
Posts: 72

Re: SAS VA Node is down

Running on a distributed environment i.e. 8 servers in the cluster.  Not sure about security scanner.  Why this would cause only one server to drop off from the cluster ?

Trusted Advisor
Posts: 1,852

Re: SAS VA Node is down

Hello @SAS_1001,

 

could you provide us more information about "going down"? Meaning:

- is the full server going to stop mode?

- are there LASR servers stopping?

- Is there any SAS service that stops?

- Is there any error ? 

- Is that the users cannot access the VA URL anymore?

- etc

 

Any detail would help us to understand. Thank you in advance.

Contributor
Posts: 72

Re: SAS VA Node is down

Posted in reply to JuanS_OCS

could you provide us more information about "going down"? Meaning:

- is the full server going to stop mode?  YES

- are there LASR servers stopping?  HEAD NODE IS DOWN SO ALL OTHERS ARE ALSO STOPPING.  

- Is there any SAS service that stops?  NOPE

- Is there any error ?    :  YES - THE ERROR IS: 

[pool-3-thread-1] ERROR [unknown] com.sas.svcs.search.index.searchIndexFeedScheduler - Execution of index feeding (for all types) scheduled task failed abruptly.

- Is that the users cannot access the VA URL anymore?  YES UNTIL THE SERVER IS RE-STARTED

- etc

Trusted Advisor
Posts: 1,852

Re: SAS VA Node is down

Now I understando, so the OS is basically shutting down abruptly.

 

I think this requires investigation, and not by SAS, but by your HW/OS admins. The reasons can be really many:

- temporature, an external signal, overuse of certain resources.... how is your server SESecurity? permissive or disabled?

 

Sending you some info.... but , really, first system admins must investigate. Not SAS Admins.

 

https://unix.stackexchange.com/questions/9819/how-to-find-out-from-the-logs-what-caused-system-shutd...

https://serverfault.com/questions/386985/how-to-investigate-unexpected-linux-server-shut-down

Contributor
Posts: 72

Re: SAS VA Node is down

Posted in reply to JuanS_OCS

Basically, when I said SAS VA Node is going down means just the LASR Server and we need to re-start it using tc-runtime script.  Server stays up but the SAS VA Services is going down.  This is happening on a weekly basis.  Hope this helps.

Trusted Advisor
Posts: 1,852

Re: SAS VA Node is down

[ Edited ]

Hello @SAS_1001,

 

thanks for the clarification. So the Web Application is the one that goes down.

 

Do you have everything in one SASServer1_1, or do you have SASServer2_1 and SASServer12_1 as well? Which one of those are going down? 

 

When they go down, is there any other service that is down as well, such as the Cache Locator or JMS service?

 

Also, my suggestion, when they go down, check those items, and try to check the logs:  server.log, errors.log of the SASServerN_1 failed, it should give additional information.

 

If you have SAS Environment Manager, you can check the historic information. My expectation is that there is a chance that GC is having a hard time and JVM Heap is at the limit. You could check also the Pools and threads open, and the config on your web application server.

 

All in all, I would like to recommend you to go through the SAS 9.4 Middle-tier Administration guide and apply the  fine Tuning guidelines for Large servers, in your Web Application Server and your Web Server and cache locator. 

 

You can start in this chapter, once done, check the other chapters:https://support.sas.com/documentation/cdl/en/appsrvtuning/69859/HTML/default/viewer.htm#p1tpeih89ioi...

 

Contributor
Posts: 72

Re: SAS VA Node is down

Posted in reply to JuanS_OCS
Thank you and appreciated !
Let me go through the document and go from there.
Ask a Question
Discussion stats
  • 8 replies
  • 116 views
  • 2 likes
  • 3 in conversation