SAS Viya 3.4 installation - RabbitMQ

[Martin5]

Hi,

im having problem with Model Studio installation since this Wednesday, before it went ok but now those errors comes up

First error is from playbook and the second one from RabbitMQ service itself (it seems that there might be newer version of systemd which behave more strict with PID files)

We are using mirrored repository, i have updated it to the latest version so as the playbooks.

TASK [rabbitmq-x64_redhat_linux_6-yum : Execute Invoke cluster setup] **************************************************************************************************************************************

fatal: [sascc]: FAILED! => {"changed": true, "cmd": ["/opt/sas/viya/home/bin/setup_rabbit_cluster", "--hostlist", "sas-cas-master", "--home", "/opt/sas/viya/home", "--config", "/opt/sas/viya/config", "--service", "sas-viya-rabbitmq-server-default", "--logfile", "/tmp/sas_setup_rabbit_cluster.log"], "delta": "0:05:04.818840", "end": "2019-08-16 09:39:27.032624", "msg": "non-zero return code", "rc": 3, "start": "2019-08-16 09:34:22.213784", "stderr": "", "stderr_lines": [], "stdout": "Fri Aug 16 09:34:22 UTC 2019 setup_rabbit_cluster *************
Fri Aug 16 09:34:22 UTC 2019 setup_rabbit_cluster Beginning SAS RabbitMQ clustering setup
Fri Aug 16 09:34:22 UTC 2019 (debug) setup_rabbit_cluster SSL is true
Fri Aug 16 09:34:22 UTC 2019 (debug) setup_rabbit_cluster CHECK_PORT is 5671
Fri Aug 16 09:34:22 UTC 2019 (debug) setup_rabbit_cluster First host in list is sas-cas-master
Fri Aug 16 09:34:22 UTC 2019 (debug) setup_rabbit_cluster First host shortname is sas-cas-master-tr3-0
Fri Aug 16 09:34:22 UTC 2019 (debug) setup_rabbit_cluster Primary Host is sas-cas-master
Fri Aug 16 09:34:22 UTC 2019 (debug) setup_rabbit_cluster Primary Short Host is sas-cas-master-tr3-0
Fri Aug 16 09:34:22 UTC 2019 (debug) setup_rabbit_cluster My Host is sas-cas-master
Fri Aug 16 09:34:22 UTC 2019 (debug) setup_rabbit_cluster I am the primary host
Fri Aug 16 09:34:22 UTC 2019 setup_rabbit_cluster Copy generated Erlang shared secret to Rabbit.
Fri Aug 16 09:34:26 UTC 2019 setup_rabbit_cluster Copying generated Erlang shared secret to Rabbit.
Fri Aug 16 09:34:26 UTC 2019 setup_rabbit_cluster Starting sas-viya-rabbitmq-server-default
Fri Aug 16 09:39:27 UTC 2019 setup_rabbit_cluster The RabbitMQ service sas-viya-rabbitmq-server-default could not be started.
Fri Aug 16 09:39:27 UTC 2019 setup_rabbit_cluster Returned status was 1
Fri Aug 16 09:39:27 UTC 2019 setup_rabbit_cluster Setup failed. Beware of independent Rabbit hosts running with different shared secrets.", "stdout_lines": 
["Fri Aug 16 09:34:22 UTC 2019 setup_rabbit_cluster *************", 
"Fri Aug 16 09:34:22 UTC 2019 setup_rabbit_cluster Beginning SAS RabbitMQ clustering setup", 
"Fri Aug 16 09:34:22 UTC 2019 (debug) setup_rabbit_cluster SSL is true", 
"Fri Aug 16 09:34:22 UTC 2019 (debug) setup_rabbit_cluster CHECK_PORT is 5671", 
"Fri Aug 16 09:34:22 UTC 2019 (debug) setup_rabbit_cluster First host in list is sas-cas-master", 
"Fri Aug 16 09:34:22 UTC 2019 (debug) setup_rabbit_cluster First host shortname is sas-cas-master-tr3-0", 
"Fri Aug 16 09:34:22 UTC 2019 (debug) setup_rabbit_cluster Primary Host is sas-cas-master", 
"Fri Aug 16 09:34:22 UTC 2019 (debug) setup_rabbit_cluster Primary Short Host is sas-cas-master-tr3-0", 
"Fri Aug 16 09:34:22 UTC 2019 (debug) setup_rabbit_cluster My Host is sas-cas-master", 
"Fri Aug 16 09:34:22 UTC 2019 (debug) setup_rabbit_cluster I am the primary host", 
"Fri Aug 16 09:34:22 UTC 2019 setup_rabbit_cluster Copy generated Erlang shared secret to Rabbit.", 
"Fri Aug 16 09:34:26 UTC 2019 setup_rabbit_cluster Copying generated Erlang shared secret to Rabbit.", 
"Fri Aug 16 09:34:26 UTC 2019 setup_rabbit_cluster Starting sas-viya-rabbitmq-server-default", 
"Fri Aug 16 09:39:27 UTC 2019 setup_rabbit_cluster The RabbitMQ service sas-viya-rabbitmq-server-default could not be started.", 
"Fri Aug 16 09:39:27 UTC 2019 setup_rabbit_cluster Returned status was 1", 
"Fri Aug 16 09:39:27 UTC 2019 setup_rabbit_cluster Setup failed. Beware of independent Rabbit hosts running with different shared secrets."]}
NO MORE HOSTS LEFT *****************************************************************************************************************************************************************************************
PLAY RECAP *************************************************************************************************************************************************************************************************

deployTarget               : ok=204  changed=94   unreachable=0    failed=0
localhost                  : ok=14   changed=1    unreachable=0    failed=0
sascc                      : ok=170  changed=54   unreachable=0    failed=1
sascw1                     : ok=150  changed=43   unreachable=0    failed=0
sascw2                     : ok=150  changed=43   unreachable=0    failed=0
sascw3                     : ok=150  changed=43   unreachable=0    failed=0
sasprog                    : ok=165  changed=50   unreachable=0    failed=0
sasstate                   : ok=173  changed=75   unreachable=0    failed=0

[root@sas-cas-master-tr3-0 sas]# systemctl status sas-viya-rabbitmq-server-default.service -l

sas-viya-rabbitmq-server-default.service - LSB: start and stop sas-rabbitmq-server server
   Loaded: loaded (/etc/rc.d/init.d/sas-viya-rabbitmq-server-default; bad; vendor preset: disabled)
   Active: activating (start) since Fri 2019-08-16 12:39:38 UTC; 17s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 106067 ExecStart=/etc/rc.d/init.d/sas-viya-rabbitmq-server-default start (code=exited, status=0/SUCCESS)
    Tasks: 1
   Memory: 1.1M
   CGroup: /system.slice/sas-viya-rabbitmq-server-default.service
           └─106731 /bin/bash /opt/sas/viya/home/sbin/rabbitmq-server


Aug 16 12:39:38 sas-cas-master systemd[1]: Starting LSB: start and stop sas-rabbitmq-server server...
Aug 16 12:39:40 sas-cas-master su[106201]: (to sasrabbitmq) root on none
Aug 16 12:39:42 sas-cas-master sas-viya-rabbitmq-server-default[106067]: sas-viya-rabbitmq-server-default service is registered
Aug 16 12:39:47 sas-cas-master su[106933]: (to sasrabbitmq) root on none
Aug 16 12:39:47 sas-cas-master su[107085]: (to sasrabbitmq) root on none
Aug 16 12:39:53 sas-cas-master su[107565]: (to sasrabbitmq) root on none
Aug 16 12:39:54 sas-cas-master sas-viya-rabbitmq-server-default[106067]: sas-viya-rabbitmq-server-default service is started
Aug 16 12:39:54 sas-cas-master systemd[1]: Refusing to accept PID outside of service control group, acquired through unsafe symlink chain: /opt/sas/viya/config/var/run/rabbitmq-server/sas-viya-rabbitmq-server-default.pid
Aug 16 12:39:54 sas-cas-master systemd[1]: Refusing to accept PID outside of service control group, acquired through unsafe symlink chain: /opt/sas/viya/config/var/run/rabbitmq-server/sas-viya-rabbitmq-server-default.pid

Thanks for any help.

[alexal]

@Martin5 ,

Please show me the output from these commands:

cat /etc/*release*
rpm -qa systemd

[Martin5]

[root@sas-cas-master-tr3-0 virk]# cat /etc/*release*
NAME="Red Hat Enterprise Linux Server"
VERSION="7.6 (Maipo)"
ID="rhel"
ID_LIKE="fedora"
VARIANT="Server"
VARIANT_ID="server"
VERSION_ID="7.6"
PRETTY_NAME="Red Hat Enterprise Linux Server 7.6 (Maipo)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:7.6:GA:server"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7"
REDHAT_BUGZILLA_PRODUCT_VERSION=7.6
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="7.6"
Red Hat Enterprise Linux Server release 7.6 (Maipo)
Red Hat Enterprise Linux Server release 7.6 (Maipo)
cpe:/o:redhat:enterprise_linux:7.6:ga:server

[root@sas-cas-master-tr3-0 virk]# rpm -qa systemd
systemd-219-67.el7_7.1.x86_64

[alexal · Accepted solution]

@Martin5 ,

systemd-219-67.el7_7.1.x86_64

There is an integration problem between RabbitMQ and systemd-219-67.el7_7.1.x86_64, so you need to downgrade systemd to version 62. This is actively being investigated, but a fix is not yet available.

[davidvazqueztds]

Hello,

I have this error: 

TASK [rabbitmq-x64_redhat_linux_6-yum : Execute Invoke cluster setup] **********
fatal: [deployTarget]: FAILED! => {"changed": true, "cmd": ["/opt/sas/viya/home/bin/setup_rabbit_cluster", "--hostlist", "localhost", "--home", "/opt/sas/viya/home", "--config", "/opt/sas/viya/config", "--service", "sas-viya-rabbitmq-server-default", "--logfile", "/tmp/sas_setup_rabbit_cluster.log"], "delta": "0:00:18.136478", "end": "2019-09-11 08:44:03.213403", "msg": "non-zero return code", "rc": 3, "start": "2019-09-11 08:43:45.076925", "stderr": "", "stderr_lines": [], "stdout": "mié sep 11 08:43:45 CEST 2019 setup_rabbit_cluster *************\nmié sep 11 08:43:45 CEST 2019 setup_rabbit_cluster Beginning SAS RabbitMQ clustering setup\nmié sep 11 08:43:45 CEST 2019 (debug) setup_rabbit_cluster SSL is true\nmié sep 11 08:43:45 CEST 2019 (debug) setup_rabbit_cluster CHECK_PORT is 5671\nmié sep 11 08:43:45 CEST 2019 (debug) setup_rabbit_cluster First host in list is localhost\nmié sep 11 08:43:45 CEST 2019 (debug) setup_rabbit_cluster First host shortname is localhost\nmié sep 11 08:43:45 CEST 2019 (debug) setup_rabbit_cluster Primary Host is localhost\nmié sep 11 08:43:45 CEST 2019 (debug) setup_rabbit_cluster Primary Short Host is localhost\nmié sep 11 08:43:45 CEST 2019 (debug) setup_rabbit_cluster My Host is localhost\nmié sep 11 08:43:45 CEST 2019 (debug) setup_rabbit_cluster I am the primary host\nmié sep 11 08:43:45 CEST 2019 setup_rabbit_cluster Copy generated Erlang shared secret to Rabbit.\nmié sep 11 08:43:50 CEST 2019 setup_rabbit_cluster Copying generated Erlang shared secret to Rabbit.\nmié sep 11 08:43:50 CEST 2019 setup_rabbit_cluster Starting sas-viya-rabbitmq-server-default\nmié sep 11 08:44:03 CEST 2019 setup_rabbit_cluster The RabbitMQ service sas-viya-rabbitmq-server-default could not be started.\nmié sep 11 08:44:03 CEST 2019 setup_rabbit_cluster Returned status was 1\nmié sep 11 08:44:03 CEST 2019 setup_rabbit_cluster Setup failed. Beware of independent Rabbit hosts running with different shared secrets.", "stdout_lines": ["mié sep 11 08:43:45 CEST 2019 setup_rabbit_cluster *************", "mié sep 11 08:43:45 CEST 2019 setup_rabbit_cluster Beginning SAS RabbitMQ clustering setup", "mié sep 11 08:43:45 CEST 2019 (debug) setup_rabbit_cluster SSL is true", "mié sep 11 08:43:45 CEST 2019 (debug) setup_rabbit_cluster CHECK_PORT is 5671", "mié sep 11 08:43:45 CEST 2019 (debug) setup_rabbit_cluster First host in list is localhost", "mié sep 11 08:43:45 CEST 2019 (debug) setup_rabbit_cluster First host shortname is localhost", "mié sep 11 08:43:45 CEST 2019 (debug) setup_rabbit_cluster Primary Host is localhost", "mié sep 11 08:43:45 CEST 2019 (debug) setup_rabbit_cluster Primary Short Host is localhost", "mié sep 11 08:43:45 CEST 2019 (debug) setup_rabbit_cluster My Host is localhost", "mié sep 11 08:43:45 CEST 2019 (debug) setup_rabbit_cluster I am the primary host", "mié sep 11 08:43:45 CEST 2019 setup_rabbit_cluster Copy generated Erlang shared secret to Rabbit.", "mié sep 11 08:43:50 CEST 2019 setup_rabbit_cluster Copying generated Erlang shared secret to Rabbit.", "mié sep 11 08:43:50 CEST 2019 setup_rabbit_cluster Starting sas-viya-rabbitmq-server-default", "mié sep 11 08:44:03 CEST 2019 setup_rabbit_cluster The RabbitMQ service sas-viya-rabbitmq-server-default could not be started.", "mié sep 11 08:44:03 CEST 2019 setup_rabbit_cluster Returned status was 1", "mié sep 11 08:44:03 CEST 2019 setup_rabbit_cluster Setup failed. Beware of independent Rabbit hosts running with different shared secrets."]}

Could it be because of the same problem? I mean, I have systemd-219-67.el7_7.1.x86_64 too....

Thanks! 

[Martin5]

Hi, it looks like you have the same problem, try to rollback to version 219-62.el7_6.7 of systemd (this one is working for us)

[davidvazqueztds]

How should I do the rollback? I tryed but I didn´t find the way... 

I tryed "yum downgrade systemd..." but it didn´t work. ¿How should I do it? 

Thank you very much!

[alexal]

@davidvazqueztds ,

We have already released the fix for systemd problem last Friday. You have to resync your local mirror (if exists), reset YUM cache and redeploy/update the software. But your problem isn't with systemd, but rather with hostname resolution.

Each machine in the deployment must have a fully qualified domain name (FQDN). To ensure that each machine in the deployment has the host name that you expect, run the hostname, hostname -f, and the hostname -s commands on each machine. If any of the machines are not named as you expect or do not have an FQDN, correct the issue and run the commands again to confirm the correction.

Primary Host is localhost\nmié sep 11 08:43:45 CEST 2019 (debug) setup_rabbit_cluster Primary Short Host is localhost

In your case, these commands return "localhost", which isn't correct.

For more information about the hostname command and its options, see the Linux man pages.

[davidvazqueztds]

I don´t have FQDN because it´s a local installation in a single machine. I will use only one machine. I used "localhost.localdomain" as a FQDN. is it valid?

hostname -s and hostname -f commands returns "ctl"

[davidvazqueztds]

Hello,

I changed the hostname, but I still get the same error:

Does anyone knows whats wrong?

Thanks again! 

Error:

TASK [rabbitmq-x64_redhat_linux_6-yum : Execute Invoke cluster setup] **********
fatal: [deployTarget]: FAILED! => {"changed": true, "cmd": ["/opt/sas/viya/home/bin/setup_rabbit_cluster", "--hostlist", "sas.tecdesoft.es", "--home", "/opt/sas/viya/home", "--config", "/opt/sas/viya/config", "--service", "sas-viya-rabbitmq-server-default", "--logfile", "/tmp/sas_setup_rabbit_cluster.log"], "delta": "0:08:01.041285", "end": "2019-09-11 13:59:34.235547", "msg": "non-zero return code", "rc": 3, "start": "2019-09-11 13:51:33.194262", "stderr": "", "stderr_lines": [], "stdout": "mié sep 11 13:51:33 CEST 2019 setup_rabbit_cluster *************\nmié sep 11 13:51:33 CEST 2019 setup_rabbit_cluster Beginning SAS RabbitMQ clustering setup\nmié sep 11 13:51:36 CEST 2019 (debug) setup_rabbit_cluster SSL is true\nmié sep 11 13:51:36 CEST 2019 (debug) setup_rabbit_cluster CHECK_PORT is 5671\nmié sep 11 13:51:36 CEST 2019 (debug) setup_rabbit_cluster First host in list is sas.tecdesoft.es\nmié sep 11 13:51:36 CEST 2019 (debug) setup_rabbit_cluster First host shortname is sas\nmié sep 11 13:51:36 CEST 2019 (debug) setup_rabbit_cluster Primary Host is sas.tecdesoft.es\nmié sep 11 13:51:36 CEST 2019 (debug) setup_rabbit_cluster Primary Short Host is sas\nmié sep 11 13:51:36 CEST 2019 (debug) setup_rabbit_cluster My Host is sas.tecdesoft.es\nmié sep 11 13:51:36 CEST 2019 (debug) setup_rabbit_cluster I am the primary host\nmié sep 11 13:51:36 CEST 2019 setup_rabbit_cluster Copy generated Erlang shared secret to Rabbit.\nmié sep 11 13:51:52 CEST 2019 setup_rabbit_cluster Shutting down sas-viya-rabbitmq-server-default to copy generated Erlang shared secret file.\nmié sep 11 13:52:01 CEST 2019 setup_rabbit_cluster Shutting down sas-viya-rabbitmq-server-default.\nmié sep 11 13:52:21 CEST 2019 (debug) setup_rabbit_cluster Shutdown of sas-viya-rabbitmq-server-default succeeded.\nmié sep 11 13:52:21 CEST 2019 setup_rabbit_cluster Copying generated Erlang shared secret to Rabbit.\nmié sep 11 13:52:21 CEST 2019 setup_rabbit_cluster Starting sas-viya-rabbitmq-server-default\nmié sep 11 13:59:34 CEST 2019 setup_rabbit_cluster The RabbitMQ service sas-viya-rabbitmq-server-default could not be started.\nmié sep 11 13:59:34 CEST 2019 setup_rabbit_cluster Returned status was 1\nmié sep 11 13:59:34 CEST 2019 setup_rabbit_cluster Setup failed. Beware of independent Rabbit hosts running with different shared secrets.", "stdout_lines": ["mié sep 11 13:51:33 CEST 2019 setup_rabbit_cluster *************", "mié sep 11 13:51:33 CEST 2019 setup_rabbit_cluster Beginning SAS RabbitMQ clustering setup", "mié sep 11 13:51:36 CEST 2019 (debug) setup_rabbit_cluster SSL is true", "mié sep 11 13:51:36 CEST 2019 (debug) setup_rabbit_cluster CHECK_PORT is 5671", "mié sep 11 13:51:36 CEST 2019 (debug) setup_rabbit_cluster First host in list is sas.tecdesoft.es", "mié sep 11 13:51:36 CEST 2019 (debug) setup_rabbit_cluster First host shortname is sas", "mié sep 11 13:51:36 CEST 2019 (debug) setup_rabbit_cluster Primary Host is sas.tecdesoft.es", "mié sep 11 13:51:36 CEST 2019 (debug) setup_rabbit_cluster Primary Short Host is sas", "mié sep 11 13:51:36 CEST 2019 (debug) setup_rabbit_cluster My Host is sas.tecdesoft.es", "mié sep 11 13:51:36 CEST 2019 (debug) setup_rabbit_cluster I am the primary host", "mié sep 11 13:51:36 CEST 2019 setup_rabbit_cluster Copy generated Erlang shared secret to Rabbit.", "mié sep 11 13:51:52 CEST 2019 setup_rabbit_cluster Shutting down sas-viya-rabbitmq-server-default to copy generated Erlang shared secret file.", "mié sep 11 13:52:01 CEST 2019 setup_rabbit_cluster Shutting down sas-viya-rabbitmq-server-default.", "mié sep 11 13:52:21 CEST 2019 (debug) setup_rabbit_cluster Shutdown of sas-viya-rabbitmq-server-default succeeded.", "mié sep 11 13:52:21 CEST 2019 setup_rabbit_cluster Copying generated Erlang shared secret to Rabbit.", "mié sep 11 13:52:21 CEST 2019 setup_rabbit_cluster Starting sas-viya-rabbitmq-server-default", "mié sep 11 13:59:34 CEST 2019 setup_rabbit_cluster The RabbitMQ service sas-viya-rabbitmq-server-default could not be started.", "mié sep 11 13:59:34 CEST 2019 setup_rabbit_cluster Returned status was 1", "mié sep 11 13:59:34 CEST 2019 setup_rabbit_cluster Setup failed. Beware of independent Rabbit hosts running with different shared secrets."]}

[alexal]

@davidvazqueztds ,

Now it looks like you have issues with systemd. Show me the output from "yum info sas-rabbitmq-server" command.

[davidvazqueztds]

Here´s the result:

[root@sas sas_viya_playbook]# yum info sas-rabbitmq-server
Complementos cargados:langpacks, product-id, search-disabled-repos,
: subscription-manager
Paquetes instalados
Nombre : sas-rabbitmq-server
Arquitectura : x86_64
Versión : 3.7.11
Lanzamiento : 20190826.1566842502511
Tamaño : 11 M
Repositorio : installed
Desde el repositorio : sas-esp-110-x64_redhat_linux_6-yum
Resumen : RabbitMQ is an implementation of AMQP, the emerging standard for
: high performance enterprise messaging. The RabbitMQ server is a
: robust and scalable implementation of an AMQP broker.
URL : http://www.sas.com
Licencia : SAS Institute Inc.
Descripción :RabbitMQ is an implementation of AMQP, the emerging standard for
: high performance enterprise messaging. The RabbitMQ server is a
: robust and scalable implementation of an AMQP broker.
: Sas-rabbitmq-server 3.7.11 20190826.1566842502511

[root@sas sas_viya_playbook]#

Thank you very much

[alexal]

@davidvazqueztds ,

Good. You have the latest version of the package with the fix. Show me the output from "ps -ef | grep -i erl" command.

[davidvazqueztds]

Her´s the result:

[root@sas sas_viya_playbook]# ps -ef | grep -i erl
sasrabb+ 26679 1 0 13:52 ? 00:00:00 /opt/sas/viya/home/lib64/erlang/erts-10.2.4/bin/epmd -daemon
root 60596 5802 0 15:05 pts/0 00:00:00 grep --color=auto -i erl
[root@sas sas_viya_playbook]#

Thank  you very much

[alexal]

@davidvazqueztds ,

Exactly what I expected to see. Please run "sudo kill -9 26679" and try to start RabbitMQ again.