SAS Viya 3.5 Single Machine Deployment Issues

[mrafiqahmed]

SAS Viya 3.5 - Single machine deployment failed with the following lines of errors

2020-11-09 16:14:16,578 p=sas u=22400 | Read vars_file '../vars.yml'
2020-11-09 16:14:16,579 p=sas u=22400 | TASK [vault-x64_redhat_linux_6-yum : Execute Wait for consul kill script to finish] ************************************************************************************
2020-11-09 16:14:16,593 p=sas u=31599 | <deployTarget> ESTABLISH LOCAL CONNECTION FOR USER: sas
2020-11-09 16:14:16,593 p=sas u=31599 | <deployTarget> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /tmp/.$USER.ansible/ansible-tmp-1604918656.5928502-86171358661829 `" && echo ansible-tmp-1604918656.5928502-86171358661829="` echo /tmp/.$USER.ansible/ansible-tmp-1604918656.5928502-86171358661829 `" ) && sleep 0'
2020-11-09 16:14:16,602 p=sas u=31599 | Using module file /usr/local/lib/python3.6/site-packages/ansible/modules/commands/command.py
2020-11-09 16:14:16,602 p=sas u=31599 | <deployTarget> PUT /root/.ansible/tmp/ansible-local-22400iyll7t_6/tmpkzeasdjh TO /tmp/.sas.ansible/ansible-tmp-1604918656.5928502-86171358661829/AnsiballZ_command.py
2020-11-09 16:14:16,603 p=sas u=31599 | <deployTarget> EXEC /bin/sh -c 'chmod u+x /tmp/.sas.ansible/ansible-tmp-1604918656.5928502-86171358661829/ /tmp/.sas.ansible/ansible-tmp-1604918656.5928502-86171358661829/AnsiballZ_command.py && sleep 0'
2020-11-09 16:14:16,609 p=sas u=31599 | <deployTarget> EXEC /bin/sh -c 'sudo -H -S -n -u root /bin/sh -c '"'"'echo BECOME-SUCCESS-fhuzzmdvtjdqekvzloaibeaiesglmfwq ; /usr/bin/python /tmp/.sas.ansible/ansible-tmp-1604918656.5928502-86171358661829/AnsiballZ_command.py'"'"' && sleep 0'

" PID TTY STAT TIME COMMAND",
"32502 ? Sl 0:15 /opt/sas/viya/home/bin/consul agent -pid-file=/var/run/sas/sas-viya-consul-default.pid -config-dir /opt/sas/viya/config/etc/consul.d -data-dir /opt/sas/viya/config/data/consul -bootstrap-expect 1 -bind  -node sasviya -datacenter viya -retry-join sasviya -server -client  127.0.0.1",
"2020-11-09 12:46:27 wait_until_consul_killed.sh main 63 [ ERROR ] 20 minutes elapsed waiting for Consul to be killed. The kill scripts likely exited with an error.",
"2020-11-09 12:46:27 wait_until_consul_killed.sh main 64 [ ERROR ] Check the kill_consul_helper log to determine the cause."

]
}
2020-11-09 12:46:28,033 p=sas u=24740 | NO MORE HOSTS LEFT ***********************************************************************************************************************************
2020-11-09 12:46:28,034 p=sas u=24740 | PLAY RECAP *******************************************************************************************************************************************
2020-11-09 12:46:28,034 p=sas u=24740 | deployTarget : ok=278 changed=64 unreachable=0 failed=1 skipped=810 rescued=0 ignored=0
2020-11-09 12:46:28,034 p=sas u=24740 | localhost : ok=11 changed=1 unreachable=0 failed=0 skipped=7 rescued=0 ignored=0

Please help

[AnandVyas]

Hi @mrafiqahmed 

What do you see in the kill_consul_helper log? Similar error was reported earlier for Viya 3.4 which was resolved by adding hostname in /etc/hosts file.

https://communities.sas.com/t5/SAS-Viya/SAS-Viya-Deployment-Error/m-p/601136

[mrafiqahmed]

I could not find that log at all.

And, I have added the hostname in /etc/hosts already; no luck

[mrafiqahmed]

For rabbitmq service, we get this error

Error: unable to perform an operation on node 'rabbit@sasviya.localdomain'. Please see diagnostics information and suggestions below.

Most common reasons for this are:

* Target node is unreachable (e.g. due to hostname resolution, TCP connection or firewall issues)
* CLI tool fails to authenticate with the server (e.g. due to CLI tool's Erlang cookie not matching that of the server)
* Target node is not running

In addition to the diagnostics info below:

* See the CLI, clustering and networking guides on https://rabbitmq.com/documentation.html to learn more
* Consult server logs on node rabbit@sasviya.localdomain
* If target node is configured to use long node names, don't forget to use --longnames with CLI tools

DIAGNOSTICS
===========

attempted to contact: ['rabbit@sasviya.localdomain']

rabbit@sasviya.localdomain:
* connected to epmd (port 4369) on sasviya.localdomain
* epmd reports node 'rabbit' uses port 25672 for inter-node and CLI tool traffic
* TCP connection succeeded but Erlang distribution failed

* TCP connection to remote host has timed out. Is the Erlang distribution using TLS?

Current node details:
* node name: 'rabbitmqcli-11265-rabbit@sasviya.localdomain'
* effective user's home directory: /opt/sas/viya/config/var/lib/rabbitmq-server/sasrabbitmq
* Erlang cookie hash: s+qk08YcclwLfGoVIrJddQ==

rabbitmq failed to respond, check if rabbitmq server is running locally

[AnandVyas · Accepted solution]

I faced a similar issue recently during a Viya SMP deployment. Are you using a FQDN on this machine? Your output for hostname -f and hostname -s should not be same.

[mrafiqahmed]

Thanks @AnandVyas  . RabbitMQ issue got resolved now.

[Chethandeepak]

hostname -f and hostname -s are not same in my server ,I'm doing installation for mpp. is there any other way to resolve it. Please help me out

[gwootton]

What issue are you encountering? Did you check the logs?

[Chethandeepak]

Hii

Greetings

encountering 

"2020-11-09 12:46:27 wait_until_consul_killed.sh main 63 [ ERROR ] 20 minutes elapsed waiting for Consul to be killed. The kill scripts likely exited with an error.",
"2020-11-09 12:46:27 wait_until_consul_killed.sh main 64 [ ERROR ] Check the kill_consul_helper log to determine the cause."

]
  

couldnt able find locaton of this log  kill_consul_helper ?

host entry in /etc/hosts/  file  is done as required and hostname -f and hostname -s are not same, still  persists same error above mentioned . could you please help me out to resolve this .

[gwootton]

The log would be in /opt/sas/viya/config/var/log/consul/default and have a name like: kill_consul_helper_2023-05-09_16-31-11.log
You might also want to check the other consul logs there.
Have you engaged technical support?

[Chethandeepak]

no there is no log like kill_consul_helper_2023-05-09_16-31-11.log in /opt/sas/viya/config/var/log/consul/default.

Have you engaged technical support?

not yet , looking to engaged technical support.

[gwootton]

Are there other logs in that path? The error is saying after telling consul to stop it did not after 20 minutes.

I would recommend engaging with SAS Technical Support.

[Chethandeepak]

is there might be any DNS related to this issue? still this error is persisting.

[gwootton]

All we know from the error is that consul is not stopping. Have you opened a technical support track? I suspect this will require they review your log files more completely.