SAS Viya 3.4 - /etc/init.d/sas-viya-spawner-default status shows sas-viya-spawner-default is dead

[Martin_Kolberg]

Hi together,

I want to share my issue with SAS Viya 3.4 and how I resolved it. Didn't find any hint over Technical Support Search or Google search.

Best Regards
Martin

############################################
# Problem sas-viya-spawner-default is dead
############################################

[sas@sasviya init.d]$ sudo ./sas-viya-spawner-default status
sas-viya-spawner-default is dead
[sas@sasviya init.d]$ sudo ./sas-viya-spawner-default restart
sas-viya-spawner-default is dead
sas-viya-spawner-default is running
[sas@sasviya init.d]$ sudo ./sas-viya-spawner-default status
sas-viya-spawner-default is dead


############################################
# Check Spawner log-files
############################################
[sas@sasviya default]$ pwd
/var/log/sas/viya/spawner/default
[sas@sasviya default]$
[sas@sasviya default]$ cat objectspawner_console_2019-01-09_10-44-04.log
Failed to initialize cradle support


[sas@sasviya default]$ cat ObjectSpawner_2019-01-09_sasviya_5740.log
Host: 'sasviya', OS: 'LIN X64', Release: '3.10.0-862.14.4.el7.x86_64', Command: '/opt/sas/spre/home/SASFoundation/utilities/bin/objspawn -dnsname localhost -configFile /opt/sas/viya/config/etc/spawner/default/spawner.cfg -logconfigloc /opt/sas/viya/config/etc/spawner/default/logconfig.xml'
2019-01-09T10:44:07,333 INFO  [00000004] :sas - SAS Object Spawner Daemon III version 9.40.20180709 is initializing.
2019-01-09T10:44:07,340 INFO  [00000004] :sas - Command line (0x0) /opt/sas/spre/home/SASFoundation/utilities/bin/objspawn
2019-01-09T10:44:07,340 INFO  [00000004] :sas -    Option (0x3) dnsname
2019-01-09T10:44:07,340 INFO  [00000004] :sas -       value (0x0) localhost
2019-01-09T10:44:07,340 INFO  [00000004] :sas -    Option (0x1) configFile
2019-01-09T10:44:07,340 INFO  [00000004] :sas -       value (0x0) /opt/sas/viya/config/etc/spawner/default/spawner.cfg
2019-01-09T10:44:07,340 INFO  [00000004] :sas -    Option (0x8000000f) logconfigloc
2019-01-09T10:44:07,341 INFO  [00000004] :sas -       value (0x0) /opt/sas/viya/config/etc/spawner/default/logconfig.xml
2019-01-09T10:44:07,360 INFO  [00000008] :sas - Objspawn is executing on host localhost (::).
2019-01-09T10:44:07,360 INFO  [00000008] :sas - Also known as:
2019-01-09T10:44:07,360 INFO  [00000008] :sas -       ::
2019-01-09T10:44:07,360 INFO  [00000008] :sas - Processing configuration file (/opt/sas/viya/config/etc/spawner/default/spawner.cfg).
2019-01-09T10:44:07,361 INFO  [00000008] :sas - Processing parsed configuration results.
2019-01-09T10:44:07,361 INFO  [00000008] :sas - Objspawn is proceeding with the configuration results processing utilizing the StudioSpawner spawner definition.
2019-01-09T10:44:07,361 ERROR [00000008] :sas - The TCP/IP tcpSockBind support routine failed with error 98 (The specified address is already in use.).
2019-01-09T10:44:07,362 ERROR [00000008] :sas - Bridge Protocol Engine Socket Access Method was unable to bind the listen socket to port 8591.
2019-01-09T10:44:07,362 WARN  [00000008] :sas - Skipping server definition (mySASObjectServer).
2019-01-09T10:44:07,362 ERROR [00000008] :sas - Objspawn encountered errors during results processing.
2019-01-09T10:44:07,362 ERROR [00000008] :sas - CLASSFACTORY kann nicht instantiiert werden.
2019-01-09T10:44:07,363 INFO  [00000009] :sas - Bridge PE SAM listen thread is exiting (0).
2019-01-09T10:44:07,363 INFO  [00000007] :sas - For the IOM thread puddle, there were 1 requests processed. The maximum number of requests queued was 0. The maximum number of worker threads was 1.
2019-01-09T10:44:07,363 INFO  [00000007] :sas - Bridge protocol engine has quiesced.
2019-01-09T10:44:07,363 INFO  [00000007] :sas - Bridge protocol engine is unloading.
2019-01-09T10:44:07,365 ERROR [00000004] :sas - Unable to initiate the IOM run-time.


#############################################
Resolution: hanging objspawn process (sasels)
#############################################
[sas@sasviya default]$ sudo netstat -anop | grep 8591
tcp        0      0 192.168.1.110:60842     192.168.1.110:8591      TIME_WAIT   -                    timewait (10,40/0/0)
tcp6       0      0 :::8591                 :::*                    LISTEN      11123/objspawn       aus (0.00/0/0)
[sas@sasviya default]$
[sas@sasviya default]$
[sas@sasviya default]$ ps -ef | grep 11123
sas      11123     1  0  2018 ?        00:00:29 /opt/sas/spre/home/SASFoundation/utilities/bin/objspawn -dnsname localhost -configFile /opt/sas/viya/config/etc/spawner/default/spawner.cfg -logconfigloc /opt/sas/viya/config/etc/spawner/default/logconfig.xml
root     11365 11123  0  2018 ?        00:00:00 sasels  7 4 8 2 1103f1
sas      12833 11122  0 11:03 pts/0    00:00:00 grep --color=auto 11123

############################################
# Kill hanging process
############################################

[sas@sasviya default]$ kill -15 11123
[sas@sasviya default]$ ps -ef | grep 11123
sas      21522 11122  0 11:05 pts/0    00:00:00 grep --color=auto 11123
[sas@sasviya default]$ sudo netstat -anop | grep 8591
tcp        0      0 192.168.1.110:38742     192.168.1.110:8591      TIME_WAIT   -                    timewait (42,02/0/0)
[sas@sasviya default]$

############################################
# Restart sas-viya-spawner-default restart
############################################

[sas@sasviya init.d]$ sudo ./sas-viya-spawner-default restart
sas-viya-spawner-default is dead
sas-viya-spawner-default is running
[sas@sasviya init.d]$ sudo ./sas-viya-spawner-default status
sas-viya-spawner-default is running


############################################
# Check again restarted objspawn process
############################################

[sas@sasviya default]$ sudo netstat -anop | grep 8591
tcp        0      0 192.168.1.110:51142     192.168.1.110:8591      TIME_WAIT   -                    timewait (21,62/0/0)
tcp6       0      0 :::8591                 :::*                    LISTEN      32660/objspawn       aus (0.00/0/0)

[sas@sasviya default]$ ps -ef | grep 32660
sas      30382 11122  0 11:16 pts/0    00:00:00 grep --color=auto 32660
sas      32660     1  0 11:08 ?        00:00:00 /opt/sas/spre/home/SASFoundation/utilities/bin/objspawn -dnsname localhost -configFile /opt/sas/viya/config/etc/spawner/default/spawner.cfg -logconfigloc /opt/sas/viya/config/etc/spawner/default/logconfig.xml
root     32663 32660  0 11:08 ?        00:00:00 sasels  7 4 8 2 7ef6c

############################################
# Done
############################################

[Debi_SAS]

Hi there,

Is it possible for you to kill all the processes and if possible reboot the system. sometimes what happens , because of some defunct processes  this kind of error might come.

Thanks,

Debi

[Martin_Kolberg]

Hi Debi,

nice to meet you again 😉

I am fine, I found out which process was hanging. I just want to mentioned and described what I did to resolve my issue.

Checking support.sas.com, I couldn't find a hint in regards of message "sas-viya-spawner-default is dead" and "Failed to initialize cradle support".

So I decided to open a thread to be able to find it over google search.

Best Regards

Martin

[Debi_SAS]

Hi Martin,

So the defunct process was the culprit?

Or anything else you did to resolve the issue.Could you please explain me how did the issue resolved .

Regards,
Debi

[Martin_Kolberg · Accepted solution]

Hi Debi,

Q: So the defunct process was the culprit?

A: YES

I checked the log-files from Object Spawner and found message regarding port already in use. Which point me to a process already running. I restart OB by sudo ./sas-viya-spawner-default restart, but it doesn't solve my issue.

Then I check for hunging processes and found a hung process from 2018. I killed it and restart OB again, which solved my issue.

#############################################
Resolution: hanging objspawn process (sasels)
#############################################
[sas@sasviya default]$ sudo netstat -anop | grep 8591
tcp        0      0 192.168.1.110:60842     192.168.1.110:8591      TIME_WAIT   -                    timewait (10,40/0/0)
tcp6       0      0 :::8591                 :::*                    LISTEN      11123/objspawn       aus (0.00/0/0)
[sas@sasviya default]$
[sas@sasviya default]$
[sas@sasviya default]$ ps -ef | grep 11123
sas      11123     1  0  2018 ?        00:00:29 /opt/sas/spre/home/SASFoundation/utilities/bin/objspawn -dnsname localhost -configFile /opt/sas/viya/config/etc/spawner/default/spawner.cfg -logconfigloc /opt/sas/viya/config/etc/spawner/default/logconfig.xml
root     11365 11123  0  2018 ?        00:00:00 sasels  7 4 8 2 1103f1
sas      12833 11122  0 11:03 pts/0    00:00:00 grep --color=auto 11123

I run kill -15 on pid 11123 to solve my issue.

Best Regards
Martin