topic Re: SAS VA Node is down in SAS Visual Analytics

SAS VA Node is down

SAS_1001 — Wed, 08 Aug 2018 19:51:38 GMT

Hello,

Our SAS VA head node is going down on a weekly basis and we need to re-start it again. We are not getting any clue from the log and wondering what could be the possibilities / reasons and what we need to do to debug this issue so this won't happen again.

Thanks in advance !

Re: SAS VA Node is down

alexal — Thu, 09 Aug 2018 10:59:36 GMT

@SAS_1001,

Your LASR server is running in distributed or non-distributed mode? Do you have any security scanners in your network?

Re: SAS VA Node is down

JuanS_OCS — Thu, 09 Aug 2018 12:50:14 GMT

Hello @SAS_1001,

could you provide us more information about "going down"? Meaning:

- is the full server going to stop mode?

- are there LASR servers stopping?

- Is there any SAS service that stops?

- Is there any error ?

- Is that the users cannot access the VA URL anymore?

- etc

Any detail would help us to understand. Thank you in advance.

Re: SAS VA Node is down

SAS_1001 — Thu, 09 Aug 2018 14:12:26 GMT

Running on a distributed environment i.e. 8 servers in the cluster. Not sure about security scanner. Why this would cause only one server to drop off from the cluster ?

Re: SAS VA Node is down

SAS_1001 — Thu, 09 Aug 2018 14:20:54 GMT

could you provide us more information about "going down"? Meaning:

- is the full server going to stop mode? YES

- are there LASR servers stopping? HEAD NODE IS DOWN SO ALL OTHERS ARE ALSO STOPPING.

- Is there any SAS service that stops? NOPE

- Is there any error ? : YES - THE ERROR IS:

[pool-3-thread-1] ERROR [unknown] com.sas.svcs.search.index.searchIndexFeedScheduler - Execution of index feeding (for all types) scheduled task failed abruptly.

- Is that the users cannot access the VA URL anymore? YES UNTIL THE SERVER IS RE-STARTED

- etc

Re: SAS VA Node is down

JuanS_OCS — Thu, 09 Aug 2018 14:27:32 GMT

Now I understando, so the OS is basically shutting down abruptly.

I think this requires investigation, and not by SAS, but by your HW/OS admins. The reasons can be really many:

- temporature, an external signal, overuse of certain resources.... how is your server SESecurity? permissive or disabled?

Sending you some info.... but , really, first system admins must investigate. Not SAS Admins.

https://unix.stackexchange.com/questions/9819/how-to-find-out-from-the-logs-what-caused-system-shutdown

https://serverfault.com/questions/386985/how-to-investigate-unexpected-linux-server-shut-down

Re: SAS VA Node is down

SAS_1001 — Thu, 09 Aug 2018 14:52:56 GMT

Basically, when I said SAS VA Node is going down means just the LASR Server and we need to re-start it using tc-runtime script. Server stays up but the SAS VA Services is going down. This is happening on a weekly basis. Hope this helps.

Re: SAS VA Node is down

JuanS_OCS — Fri, 10 Aug 2018 10:46:04 GMT

Hello @SAS_1001,

thanks for the clarification. So the Web Application is the one that goes down.

Do you have everything in one SASServer1_1, or do you have SASServer2_1 and SASServer12_1 as well? Which one of those are going down?

When they go down, is there any other service that is down as well, such as the Cache Locator or JMS service?

Also, my suggestion, when they go down, check those items, and try to check the logs: server.log, errors.log of the SASServerN_1 failed, it should give additional information.

If you have SAS Environment Manager, you can check the historic information. My expectation is that there is a chance that GC is having a hard time and JVM Heap is at the limit. You could check also the Pools and threads open, and the config on your web application server.

All in all, I would like to recommend you to go through the SAS 9.4 Middle-tier Administration guide and apply the fine Tuning guidelines for Large servers, in your Web Application Server and your Web Server and cache locator.

You can start in this chapter, once done, check the other chapters:https://support.sas.com/documentation/cdl/en/appsrvtuning/69859/HTML/default/viewer.htm#p1tpeih89ioi0in1pc7bg9cyqzuo.htm

Re: SAS VA Node is down

SAS_1001 — Fri, 10 Aug 2018 23:01:09 GMT

Thank you and appreciated !
Let me go through the document and go from there.

Re: SAS VA Node is down

SAS_1001 — Fri, 24 Aug 2018 16:29:45 GMT

Seems the issue is resolved. Thank you and appreciated !