BookmarkSubscribeRSS Feed
🔒 This topic is solved and locked. Need further help from the community? Please sign in and ask a new question.
SAS_1001
Obsidian | Level 7

Hello,

Our SAS VA head node is going down on a weekly basis and we need to re-start it again. We are not getting any clue from the log and wondering what could be the possibilities / reasons and what we need to do to debug this issue so this won't happen again.

Thanks in advance !

1 ACCEPTED SOLUTION

Accepted Solutions
JuanS_OCS
Amethyst | Level 16

Hello @SAS_1001,

 

thanks for the clarification. So the Web Application is the one that goes down.

 

Do you have everything in one SASServer1_1, or do you have SASServer2_1 and SASServer12_1 as well? Which one of those are going down? 

 

When they go down, is there any other service that is down as well, such as the Cache Locator or JMS service?

 

Also, my suggestion, when they go down, check those items, and try to check the logs:  server.log, errors.log of the SASServerN_1 failed, it should give additional information.

 

If you have SAS Environment Manager, you can check the historic information. My expectation is that there is a chance that GC is having a hard time and JVM Heap is at the limit. You could check also the Pools and threads open, and the config on your web application server.

 

All in all, I would like to recommend you to go through the SAS 9.4 Middle-tier Administration guide and apply the  fine Tuning guidelines for Large servers, in your Web Application Server and your Web Server and cache locator. 

 

You can start in this chapter, once done, check the other chapters:https://support.sas.com/documentation/cdl/en/appsrvtuning/69859/HTML/default/viewer.htm#p1tpeih89ioi...

 

View solution in original post

9 REPLIES 9
alexal
SAS Employee

@SAS_1001,

 

Your LASR server is running in distributed or non-distributed mode? Do you have any security scanners in your network?

SAS_1001
Obsidian | Level 7

Running on a distributed environment i.e. 8 servers in the cluster.  Not sure about security scanner.  Why this would cause only one server to drop off from the cluster ?

JuanS_OCS
Amethyst | Level 16

Hello @SAS_1001,

 

could you provide us more information about "going down"? Meaning:

- is the full server going to stop mode?

- are there LASR servers stopping?

- Is there any SAS service that stops?

- Is there any error ? 

- Is that the users cannot access the VA URL anymore?

- etc

 

Any detail would help us to understand. Thank you in advance.

SAS_1001
Obsidian | Level 7

could you provide us more information about "going down"? Meaning:

- is the full server going to stop mode?  YES

- are there LASR servers stopping?  HEAD NODE IS DOWN SO ALL OTHERS ARE ALSO STOPPING.  

- Is there any SAS service that stops?  NOPE

- Is there any error ?    :  YES - THE ERROR IS: 

[pool-3-thread-1] ERROR [unknown] com.sas.svcs.search.index.searchIndexFeedScheduler - Execution of index feeding (for all types) scheduled task failed abruptly.

- Is that the users cannot access the VA URL anymore?  YES UNTIL THE SERVER IS RE-STARTED

- etc

JuanS_OCS
Amethyst | Level 16

Now I understando, so the OS is basically shutting down abruptly.

 

I think this requires investigation, and not by SAS, but by your HW/OS admins. The reasons can be really many:

- temporature, an external signal, overuse of certain resources.... how is your server SESecurity? permissive or disabled?

 

Sending you some info.... but , really, first system admins must investigate. Not SAS Admins.

 

https://unix.stackexchange.com/questions/9819/how-to-find-out-from-the-logs-what-caused-system-shutd...

https://serverfault.com/questions/386985/how-to-investigate-unexpected-linux-server-shut-down

SAS_1001
Obsidian | Level 7

Basically, when I said SAS VA Node is going down means just the LASR Server and we need to re-start it using tc-runtime script.  Server stays up but the SAS VA Services is going down.  This is happening on a weekly basis.  Hope this helps.

JuanS_OCS
Amethyst | Level 16

Hello @SAS_1001,

 

thanks for the clarification. So the Web Application is the one that goes down.

 

Do you have everything in one SASServer1_1, or do you have SASServer2_1 and SASServer12_1 as well? Which one of those are going down? 

 

When they go down, is there any other service that is down as well, such as the Cache Locator or JMS service?

 

Also, my suggestion, when they go down, check those items, and try to check the logs:  server.log, errors.log of the SASServerN_1 failed, it should give additional information.

 

If you have SAS Environment Manager, you can check the historic information. My expectation is that there is a chance that GC is having a hard time and JVM Heap is at the limit. You could check also the Pools and threads open, and the config on your web application server.

 

All in all, I would like to recommend you to go through the SAS 9.4 Middle-tier Administration guide and apply the  fine Tuning guidelines for Large servers, in your Web Application Server and your Web Server and cache locator. 

 

You can start in this chapter, once done, check the other chapters:https://support.sas.com/documentation/cdl/en/appsrvtuning/69859/HTML/default/viewer.htm#p1tpeih89ioi...

 

SAS_1001
Obsidian | Level 7
Thank you and appreciated !
Let me go through the document and go from there.
SAS_1001
Obsidian | Level 7
Seems the issue is resolved. Thank you and appreciated !

sas-innovate-2024.png

Join us for SAS Innovate April 16-19 at the Aria in Las Vegas. Bring the team and save big with our group pricing for a limited time only.

Pre-conference courses and tutorials are filling up fast and are always a sellout. Register today to reserve your seat.

 

Register now!

Tips for filtering data sources in SAS Visual Analytics

See how to use one filter for multiple data sources by mapping your data from SAS’ Alexandria McCall.

Find more tutorials on the SAS Users YouTube channel.

Discussion stats
  • 9 replies
  • 1318 views
  • 3 likes
  • 3 in conversation