Object Spawner Restart Script

SriLaxman — Wed, 20 Jan 2021 07:50:52 GMT

Our SAS Grid nodes automatically closes and does not accept new jobs when the server memory reaches the threshold. During this time the Object Spawners gets hung since it continuously tries to submit job to the Grid nodes and fails. I am using the below script to restart the Object Spawners whenever it gets hung. Thought sharing this to other administrators could be helpful.

#!/bin/bash
# Configuration
# Description
# This ObjectSpawn_restart.sh script is used to restart the Object Spawners across multiple Grid nodes sequentially without any outage.
Machine=$(/bin/hostname)
case "${Machine}" in
        Gridnodes*)                  # SAS Grid TEST Environment
              servers=(Gridnode1 Gridnode2)     # Gridnodes where Object Spawnwers run
              export ScriptDir="/../../../config/Lev1/ObjectSpawnerGrid" # ObjectSpawner Directory
                ;;
        Gridnodes*)                  # SAS Grid DEV Environment
                servers=(Gridnode1 Gridnode2 Gridnode3 ..... Gridnoden)
              export ScriptDir="/../../..//config/Lev1/ObjectSpawnerGrid"
                ;;
        Gridnodes*)                  # SAS GRID PROD Environment
                servers=(Gridnode1 Gridnode2 Gridnode3)
              export ScriptDir="/../../../config/Lev1/ObjectSpawnerGrid"
                ;;

esac

# Configuration.
SSH="/usr/share/centrifydc/bin/ssh"
for num in ${servers[@]}; do
        processids=$(${SSH} ${num} ps -ef | grep ${ScriptDir} | grep -v grep | awk '{print $2}')
        count=`echo ${processids[@]} | wc -w`
        if [ ${count} -ne 0 ]; then
${SSH} ${num} kill -9 ${processids[@]}
        fi
sleep 3
        ${SSH} -tt ${num} nohup ${ScriptDir}/ObjectSpawner.sh start & >/dev/null 2>&1
        >/dev/null 2>&1
sleep 3
        ${SSH} -tt ${num} echo New PID: `cat ${ScriptDir}/server.${Machine}.pid`

done

Re: Object Spawner Restart Script

sandeeppajni2 — Thu, 21 Jan 2021 12:51:09 GMT

Hi @SriLaxman,

Thanks for sharing the script. Also, have you tried considering High Availability features of SAS to do so which will take care of additional components including Object Spawner?

Re: Object Spawner Restart Script

JuanS_OCS — Tue, 26 Jan 2021 14:03:29 GMT

Thank you for the share @SriLaxman , I like the simplicity of the script.

I adhere to @sandeeppajni2 recommendation. I would always recommend to implement High Availability, an approach supported by SAS Technical Support, and would give you an approach with some additional nice features for the user and the administrator.

Is there any particular reason why HA is not implemented in your SAS Grid Manager platform?

Re: Object Spawner Restart Script

SriLaxman — Wed, 27 Jan 2021 04:58:22 GMT

We have a cluster of 8 Grid Nodes and 3 Metadata Servers. We don't have RTM to enable to HA.

topic Re: Object Spawner Restart Script in Administration and Deployment

Object Spawner Restart Script

Re: Object Spawner Restart Script

Re: Object Spawner Restart Script

Re: Object Spawner Restart Script