- Mark as New
- Bookmark
- Subscribe
- Mute
- RSS Feed
- Permalink
- Report Inappropriate Content
Hello,
Our SAS VA head node is going down on a weekly basis and we need to re-start it again. We are not getting any clue from the log and wondering what could be the possibilities / reasons and what we need to do to debug this issue so this won't happen again.
Thanks in advance !
Accepted Solutions
- Mark as New
- Bookmark
- Subscribe
- Mute
- RSS Feed
- Permalink
- Report Inappropriate Content
Hello @SAS_1001,
thanks for the clarification. So the Web Application is the one that goes down.
Do you have everything in one SASServer1_1, or do you have SASServer2_1 and SASServer12_1 as well? Which one of those are going down?
When they go down, is there any other service that is down as well, such as the Cache Locator or JMS service?
Also, my suggestion, when they go down, check those items, and try to check the logs: server.log, errors.log of the SASServerN_1 failed, it should give additional information.
If you have SAS Environment Manager, you can check the historic information. My expectation is that there is a chance that GC is having a hard time and JVM Heap is at the limit. You could check also the Pools and threads open, and the config on your web application server.
All in all, I would like to recommend you to go through the SAS 9.4 Middle-tier Administration guide and apply the fine Tuning guidelines for Large servers, in your Web Application Server and your Web Server and cache locator.
You can start in this chapter, once done, check the other chapters:https://support.sas.com/documentation/cdl/en/appsrvtuning/69859/HTML/default/viewer.htm#p1tpeih89ioi...
- Mark as New
- Bookmark
- Subscribe
- Mute
- RSS Feed
- Permalink
- Report Inappropriate Content
Your LASR server is running in distributed or non-distributed mode? Do you have any security scanners in your network?
- Mark as New
- Bookmark
- Subscribe
- Mute
- RSS Feed
- Permalink
- Report Inappropriate Content
Running on a distributed environment i.e. 8 servers in the cluster. Not sure about security scanner. Why this would cause only one server to drop off from the cluster ?
- Mark as New
- Bookmark
- Subscribe
- Mute
- RSS Feed
- Permalink
- Report Inappropriate Content
Hello @SAS_1001,
could you provide us more information about "going down"? Meaning:
- is the full server going to stop mode?
- are there LASR servers stopping?
- Is there any SAS service that stops?
- Is there any error ?
- Is that the users cannot access the VA URL anymore?
- etc
Any detail would help us to understand. Thank you in advance.
- Mark as New
- Bookmark
- Subscribe
- Mute
- RSS Feed
- Permalink
- Report Inappropriate Content
could you provide us more information about "going down"? Meaning:
- is the full server going to stop mode? YES
- are there LASR servers stopping? HEAD NODE IS DOWN SO ALL OTHERS ARE ALSO STOPPING.
- Is there any SAS service that stops? NOPE
- Is there any error ? : YES - THE ERROR IS:
[pool-3-thread-1] ERROR [unknown] com.sas.svcs.search.index.searchIndexFeedScheduler - Execution of index feeding (for all types) scheduled task failed abruptly.
- Is that the users cannot access the VA URL anymore? YES UNTIL THE SERVER IS RE-STARTED
- etc
- Mark as New
- Bookmark
- Subscribe
- Mute
- RSS Feed
- Permalink
- Report Inappropriate Content
Now I understando, so the OS is basically shutting down abruptly.
I think this requires investigation, and not by SAS, but by your HW/OS admins. The reasons can be really many:
- temporature, an external signal, overuse of certain resources.... how is your server SESecurity? permissive or disabled?
Sending you some info.... but , really, first system admins must investigate. Not SAS Admins.
https://serverfault.com/questions/386985/how-to-investigate-unexpected-linux-server-shut-down
- Mark as New
- Bookmark
- Subscribe
- Mute
- RSS Feed
- Permalink
- Report Inappropriate Content
Basically, when I said SAS VA Node is going down means just the LASR Server and we need to re-start it using tc-runtime script. Server stays up but the SAS VA Services is going down. This is happening on a weekly basis. Hope this helps.
- Mark as New
- Bookmark
- Subscribe
- Mute
- RSS Feed
- Permalink
- Report Inappropriate Content
Hello @SAS_1001,
thanks for the clarification. So the Web Application is the one that goes down.
Do you have everything in one SASServer1_1, or do you have SASServer2_1 and SASServer12_1 as well? Which one of those are going down?
When they go down, is there any other service that is down as well, such as the Cache Locator or JMS service?
Also, my suggestion, when they go down, check those items, and try to check the logs: server.log, errors.log of the SASServerN_1 failed, it should give additional information.
If you have SAS Environment Manager, you can check the historic information. My expectation is that there is a chance that GC is having a hard time and JVM Heap is at the limit. You could check also the Pools and threads open, and the config on your web application server.
All in all, I would like to recommend you to go through the SAS 9.4 Middle-tier Administration guide and apply the fine Tuning guidelines for Large servers, in your Web Application Server and your Web Server and cache locator.
You can start in this chapter, once done, check the other chapters:https://support.sas.com/documentation/cdl/en/appsrvtuning/69859/HTML/default/viewer.htm#p1tpeih89ioi...
- Mark as New
- Bookmark
- Subscribe
- Mute
- RSS Feed
- Permalink
- Report Inappropriate Content
Let me go through the document and go from there.
- Mark as New
- Bookmark
- Subscribe
- Mute
- RSS Feed
- Permalink
- Report Inappropriate Content