BookmarkSubscribeRSS Feed
🔒 This topic is solved and locked. Need further help from the community? Please sign in and ask a new question.
Martin_Kolberg
Obsidian | Level 7

Hi together,

 

I want to share my issue with SAS Viya 3.4 and how I resolved it. Didn't find any hint over Technical Support Search or Google search.

 

Best Regards
Martin

 

############################################
# Problem sas-viya-spawner-default is dead
############################################

[sas@sasviya init.d]$ sudo ./sas-viya-spawner-default status
sas-viya-spawner-default is dead
[sas@sasviya init.d]$ sudo ./sas-viya-spawner-default restart
sas-viya-spawner-default is dead
sas-viya-spawner-default is running
[sas@sasviya init.d]$ sudo ./sas-viya-spawner-default status
sas-viya-spawner-default is dead


############################################
# Check Spawner log-files
############################################
[sas@sasviya default]$ pwd
/var/log/sas/viya/spawner/default
[sas@sasviya default]$
[sas@sasviya default]$ cat objectspawner_console_2019-01-09_10-44-04.log
Failed to initialize cradle support


[sas@sasviya default]$ cat ObjectSpawner_2019-01-09_sasviya_5740.log
Host: 'sasviya', OS: 'LIN X64', Release: '3.10.0-862.14.4.el7.x86_64', Command: '/opt/sas/spre/home/SASFoundation/utilities/bin/objspawn -dnsname localhost -configFile /opt/sas/viya/config/etc/spawner/default/spawner.cfg -logconfigloc /opt/sas/viya/config/etc/spawner/default/logconfig.xml'
2019-01-09T10:44:07,333 INFO  [00000004] :sas - SAS Object Spawner Daemon III version 9.40.20180709 is initializing.
2019-01-09T10:44:07,340 INFO  [00000004] :sas - Command line (0x0) /opt/sas/spre/home/SASFoundation/utilities/bin/objspawn
2019-01-09T10:44:07,340 INFO  [00000004] :sas -    Option (0x3) dnsname
2019-01-09T10:44:07,340 INFO  [00000004] :sas -       value (0x0) localhost
2019-01-09T10:44:07,340 INFO  [00000004] :sas -    Option (0x1) configFile
2019-01-09T10:44:07,340 INFO  [00000004] :sas -       value (0x0) /opt/sas/viya/config/etc/spawner/default/spawner.cfg
2019-01-09T10:44:07,340 INFO  [00000004] :sas -    Option (0x8000000f) logconfigloc
2019-01-09T10:44:07,341 INFO  [00000004] :sas -       value (0x0) /opt/sas/viya/config/etc/spawner/default/logconfig.xml
2019-01-09T10:44:07,360 INFO  [00000008] :sas - Objspawn is executing on host localhost (::).
2019-01-09T10:44:07,360 INFO  [00000008] :sas - Also known as:
2019-01-09T10:44:07,360 INFO  [00000008] :sas -       ::
2019-01-09T10:44:07,360 INFO  [00000008] :sas - Processing configuration file (/opt/sas/viya/config/etc/spawner/default/spawner.cfg).
2019-01-09T10:44:07,361 INFO  [00000008] :sas - Processing parsed configuration results.
2019-01-09T10:44:07,361 INFO  [00000008] :sas - Objspawn is proceeding with the configuration results processing utilizing the StudioSpawner spawner definition.
2019-01-09T10:44:07,361 ERROR [00000008] :sas - The TCP/IP tcpSockBind support routine failed with error 98 (The specified address is already in use.).
2019-01-09T10:44:07,362 ERROR [00000008] :sas - Bridge Protocol Engine Socket Access Method was unable to bind the listen socket to port 8591.
2019-01-09T10:44:07,362 WARN  [00000008] :sas - Skipping server definition (mySASObjectServer).
2019-01-09T10:44:07,362 ERROR [00000008] :sas - Objspawn encountered errors during results processing.
2019-01-09T10:44:07,362 ERROR [00000008] :sas - CLASSFACTORY kann nicht instantiiert werden.
2019-01-09T10:44:07,363 INFO  [00000009] :sas - Bridge PE SAM listen thread is exiting (0).
2019-01-09T10:44:07,363 INFO  [00000007] :sas - For the IOM thread puddle, there were 1 requests processed. The maximum number of requests queued was 0. The maximum number of worker threads was 1.
2019-01-09T10:44:07,363 INFO  [00000007] :sas - Bridge protocol engine has quiesced.
2019-01-09T10:44:07,363 INFO  [00000007] :sas - Bridge protocol engine is unloading.
2019-01-09T10:44:07,365 ERROR [00000004] :sas - Unable to initiate the IOM run-time.


#############################################
Resolution: hanging objspawn process (sasels)
#############################################
[sas@sasviya default]$ sudo netstat -anop | grep 8591
tcp        0      0 192.168.1.110:60842     192.168.1.110:8591      TIME_WAIT   -                    timewait (10,40/0/0)
tcp6       0      0 :::8591                 :::*                    LISTEN      11123/objspawn       aus (0.00/0/0)
[sas@sasviya default]$
[sas@sasviya default]$
[sas@sasviya default]$ ps -ef | grep 11123
sas      11123     1  0  2018 ?        00:00:29 /opt/sas/spre/home/SASFoundation/utilities/bin/objspawn -dnsname localhost -configFile /opt/sas/viya/config/etc/spawner/default/spawner.cfg -logconfigloc /opt/sas/viya/config/etc/spawner/default/logconfig.xml
root     11365 11123  0  2018 ?        00:00:00 sasels  7 4 8 2 1103f1
sas      12833 11122  0 11:03 pts/0    00:00:00 grep --color=auto 11123

############################################
# Kill hanging process
############################################

[sas@sasviya default]$ kill -15 11123
[sas@sasviya default]$ ps -ef | grep 11123
sas      21522 11122  0 11:05 pts/0    00:00:00 grep --color=auto 11123
[sas@sasviya default]$ sudo netstat -anop | grep 8591
tcp        0      0 192.168.1.110:38742     192.168.1.110:8591      TIME_WAIT   -                    timewait (42,02/0/0)
[sas@sasviya default]$

############################################
# Restart sas-viya-spawner-default restart
############################################

[sas@sasviya init.d]$ sudo ./sas-viya-spawner-default restart
sas-viya-spawner-default is dead
sas-viya-spawner-default is running
[sas@sasviya init.d]$ sudo ./sas-viya-spawner-default status
sas-viya-spawner-default is running


############################################
# Check again restarted objspawn process
############################################

[sas@sasviya default]$ sudo netstat -anop | grep 8591
tcp        0      0 192.168.1.110:51142     192.168.1.110:8591      TIME_WAIT   -                    timewait (21,62/0/0)
tcp6       0      0 :::8591                 :::*                    LISTEN      32660/objspawn       aus (0.00/0/0)

[sas@sasviya default]$ ps -ef | grep 32660
sas      30382 11122  0 11:16 pts/0    00:00:00 grep --color=auto 32660
sas      32660     1  0 11:08 ?        00:00:00 /opt/sas/spre/home/SASFoundation/utilities/bin/objspawn -dnsname localhost -configFile /opt/sas/viya/config/etc/spawner/default/spawner.cfg -logconfigloc /opt/sas/viya/config/etc/spawner/default/logconfig.xml
root     32663 32660  0 11:08 ?        00:00:00 sasels  7 4 8 2 7ef6c

############################################
# Done
############################################

1 ACCEPTED SOLUTION

Accepted Solutions
Martin_Kolberg
Obsidian | Level 7

Hi Debi,

 

Q: So the defunct process was the culprit?

A: YES

 

I checked the log-files from Object Spawner and found message regarding port already in use. Which point me to a process already running. I restart OB by sudo ./sas-viya-spawner-default restart, but it doesn't solve my issue.


Then I check for hunging processes and found a hung process from 2018. I killed it and restart OB again, which solved my issue.

 

#############################################
Resolution: hanging objspawn process (sasels)
#############################################
[sas@sasviya default]$ sudo netstat -anop | grep 8591
tcp        0      0 192.168.1.110:60842     192.168.1.110:8591      TIME_WAIT   -                    timewait (10,40/0/0)
tcp6       0      0 :::8591                 :::*                    LISTEN      11123/objspawn       aus (0.00/0/0)
[sas@sasviya default]$
[sas@sasviya default]$
[sas@sasviya default]$ ps -ef | grep 11123
sas      11123     1  0  2018 ?        00:00:29 /opt/sas/spre/home/SASFoundation/utilities/bin/objspawn -dnsname localhost -configFile /opt/sas/viya/config/etc/spawner/default/spawner.cfg -logconfigloc /opt/sas/viya/config/etc/spawner/default/logconfig.xml
root     11365 11123  0  2018 ?        00:00:00 sasels  7 4 8 2 1103f1
sas      12833 11122  0 11:03 pts/0    00:00:00 grep --color=auto 11123

 

I run kill -15 on pid 11123 to solve my issue.

 

Best Regards
Martin

View solution in original post

4 REPLIES 4
Debi_SAS
Fluorite | Level 6

Hi there,

 

Is it possible for you to kill all the processes and if possible reboot the system. sometimes what happens , because of some defunct processes  this kind of error might come.

 

Thanks,

Debi

Martin_Kolberg
Obsidian | Level 7

Hi Debi,

 

nice to meet you again 😉

 

I am fine, I found out which process was hanging. I just want to mentioned and described what I did to resolve my issue.

 

Checking support.sas.com, I couldn't find a hint in regards of message "sas-viya-spawner-default is dead" and "Failed to initialize cradle support".

 

So I decided to open a thread to be able to find it over google search.

 

Best Regards

Martin

Debi_SAS
Fluorite | Level 6
Hi Martin,

So the defunct process was the culprit?

Or anything else you did to resolve the issue.Could you please explain me how did the issue resolved .

Regards,
Debi
Martin_Kolberg
Obsidian | Level 7

Hi Debi,

 

Q: So the defunct process was the culprit?

A: YES

 

I checked the log-files from Object Spawner and found message regarding port already in use. Which point me to a process already running. I restart OB by sudo ./sas-viya-spawner-default restart, but it doesn't solve my issue.


Then I check for hunging processes and found a hung process from 2018. I killed it and restart OB again, which solved my issue.

 

#############################################
Resolution: hanging objspawn process (sasels)
#############################################
[sas@sasviya default]$ sudo netstat -anop | grep 8591
tcp        0      0 192.168.1.110:60842     192.168.1.110:8591      TIME_WAIT   -                    timewait (10,40/0/0)
tcp6       0      0 :::8591                 :::*                    LISTEN      11123/objspawn       aus (0.00/0/0)
[sas@sasviya default]$
[sas@sasviya default]$
[sas@sasviya default]$ ps -ef | grep 11123
sas      11123     1  0  2018 ?        00:00:29 /opt/sas/spre/home/SASFoundation/utilities/bin/objspawn -dnsname localhost -configFile /opt/sas/viya/config/etc/spawner/default/spawner.cfg -logconfigloc /opt/sas/viya/config/etc/spawner/default/logconfig.xml
root     11365 11123  0  2018 ?        00:00:00 sasels  7 4 8 2 1103f1
sas      12833 11122  0 11:03 pts/0    00:00:00 grep --color=auto 11123

 

I run kill -15 on pid 11123 to solve my issue.

 

Best Regards
Martin

suga badge.PNGThe SAS Users Group for Administrators (SUGA) is open to all SAS administrators and architects who install, update, manage or maintain a SAS deployment. 

Join SUGA 

Get Started with SAS Information Catalog in SAS Viya

SAS technical trainer Erin Winters shows you how to explore assets, create new data discovery agents, schedule data discovery agents, and much more.

Find more tutorials on the SAS Users YouTube channel.

Discussion stats
  • 4 replies
  • 3962 views
  • 2 likes
  • 2 in conversation