BookmarkSubscribeRSS Feed
SriLaxman
Fluorite | Level 6

Our SAS Grid nodes automatically closes and does not accept new jobs when the server memory reaches the threshold. During this time the Object Spawners gets hung since it continuously tries to submit job to the Grid nodes and fails. I am using the below script to restart the Object Spawners whenever it gets hung. Thought sharing this to other administrators could be helpful.

 

#!/bin/bash
# Configuration
# Description
# This ObjectSpawn_restart.sh script is used to restart the Object Spawners across multiple Grid nodes sequentially without any outage.
Machine=$(/bin/hostname)
case "${Machine}" in
        Gridnodes*)                  # SAS Grid TEST Environment
              servers=(Gridnode1 Gridnode2)     # Gridnodes where Object Spawnwers run
              export  ScriptDir="/../../../config/Lev1/ObjectSpawnerGrid" # ObjectSpawner Directory
                ;;
        Gridnodes*)                  # SAS Grid DEV Environment
                servers=(Gridnode1 Gridnode2 Gridnode3 ..... Gridnoden)
              export  ScriptDir="/../../..//config/Lev1/ObjectSpawnerGrid"
                ;;
        Gridnodes*)                  # SAS GRID PROD Environment
                servers=(Gridnode1 Gridnode2 Gridnode3)
              export  ScriptDir="/../../../config/Lev1/ObjectSpawnerGrid"
                ;;

esac

 


# Configuration.
SSH="/usr/share/centrifydc/bin/ssh"
for num in ${servers[@]}; do
        processids=$(${SSH} ${num} ps -ef | grep ${ScriptDir} | grep -v grep | awk '{print $2}')
        count=`echo ${processids[@]} | wc -w`
        if [ ${count} -ne 0 ]; then
  ${SSH} ${num} kill -9 ${processids[@]}
        fi
sleep 3
        ${SSH} -tt ${num} nohup ${ScriptDir}/ObjectSpawner.sh start & >/dev/null 2>&1
        >/dev/null 2>&1
sleep 3
        ${SSH} -tt ${num} echo New PID: `cat ${ScriptDir}/server.${Machine}.pid`

done

 

3 REPLIES 3
sandeeppajni2
Obsidian | Level 7

Hi @SriLaxman,

Thanks for sharing the script. Also, have you tried considering High Availability features of SAS to do so which will take care of additional components including Object Spawner?

JuanS_OCS
Amethyst | Level 16

Thank you for the share @SriLaxman , I like the simplicity of the script.

 

I adhere to @sandeeppajni2 recommendation. I would always recommend to implement High Availability, an approach supported by SAS Technical Support, and would give you an approach with some additional nice features for the user and the administrator.

 

Is there any particular reason why HA is not implemented in your SAS Grid Manager platform?

SriLaxman
Fluorite | Level 6

We have a cluster of 8 Grid Nodes and 3 Metadata Servers. We don't have RTM to enable to HA.

suga badge.PNGThe SAS Users Group for Administrators (SUGA) is open to all SAS administrators and architects who install, update, manage or maintain a SAS deployment. 

Join SUGA 

Get Started with SAS Information Catalog in SAS Viya

SAS technical trainer Erin Winters shows you how to explore assets, create new data discovery agents, schedule data discovery agents, and much more.

Find more tutorials on the SAS Users YouTube channel.

Discussion stats
  • 3 replies
  • 802 views
  • 3 likes
  • 3 in conversation