sas-viya-rabbitmq-server-default could not be started during SAS Viya 3.4 deployment

[DanBondarneko]

Hello,

I am attempting a fresh SAS Viya 3.4 installation on a server and I get the following error:

fatal: [deployTarget]: FAILED! => {
"changed": true,
"cmd": [
"/opt/sas/viya/home/bin/setup_rabbit_cluster",
"--hostlist",
"hegemone",
"--home",
"/opt/sas/viya/home",
"--config",
"/opt/sas/viya/config",
"--service",
"sas-viya-rabbitmq-server-default",
"--logfile",
"/tmp/sas_setup_rabbit_cluster.log"
],
"delta": "0:00:23.486520",
"end": "2019-10-30 15:16:07.540859",
"invocation": {
"module_args": {
"_raw_params": "/opt/sas/viya/home/bin/setup_rabbit_cluster --hostlist 'hegemone' --home /opt/sas/viya/home --config /opt/sas/viya/config --service sas-viya-rabbitmq-server-default --logfile /tmp/sas_setup_rabbit_cluster.log",
"_uses_shell": false,
"argv": null,
"chdir": null,
"creates": null,
"executable": null,
"removes": null,
"stdin": null,
"warn": true
}
},
"msg": "non-zero return code",
"rc": 3,
"start": "2019-10-30 15:15:44.054339",
"stderr": "",
"stderr_lines": [],
"stdout": "Wed Oct 30 15:15:44 UTC 2019 setup_rabbit_cluster *************\nWed Oct 30 15:15:44 UTC 2019 setup_rabbit_cluster Beginning SAS RabbitMQ clustering setup\nWed Oct 30 15:15:44 UTC 2019 (debug) setup_rabbit_cluster SSL is true\nWed Oct 30 15:15:44 UTC 2019 (debug) setup_rabbit_cluster CHECK_PORT is 5671\nWed Oct 30 15:15:44 UTC 2019 (debug) setup_rabbit_cluster First host in list is hegemone\nWed Oct 30 15:15:44 UTC 2019 (debug) setup_rabbit_cluster First host shortname is hegemone\nWed Oct 30 15:15:44 UTC 2019 (debug) setup_rabbit_cluster Primary Host is hegemone\nWed Oct 30 15:15:44 UTC 2019 (debug) setup_rabbit_cluster Primary Short Host is hegemone\nWed Oct 30 15:15:44 UTC 2019 (debug) setup_rabbit_cluster My Host is hegemone\nWed Oct 30 15:15:44 UTC 2019 (debug) setup_rabbit_cluster I am the primary host\nWed Oct 30 15:15:44 UTC 2019 setup_rabbit_cluster Copy generated Erlang shared secret to Rabbit.\nWed Oct 30 15:15:47 UTC 2019 setup_rabbit_cluster Copying generated Erlang shared secret to Rabbit.\nWed Oct 30 15:15:47 UTC 2019 setup_rabbit_cluster Starting sas-viya-rabbitmq-server-default\nWed Oct 30 15:16:07 UTC 2019 setup_rabbit_cluster The RabbitMQ service sas-viya-rabbitmq-server-default could not be started.\nWed Oct 30 15:16:07 UTC 2019 setup_rabbit_cluster Returned status was 1\nWed Oct 30 15:16:07 UTC 2019 setup_rabbit_cluster Setup failed. Beware of independent Rabbit hosts running with different shared secrets.",
"stdout_lines": [
"Wed Oct 30 15:15:44 UTC 2019 setup_rabbit_cluster *************",
"Wed Oct 30 15:15:44 UTC 2019 setup_rabbit_cluster Beginning SAS RabbitMQ clustering setup",
"Wed Oct 30 15:15:44 UTC 2019 (debug) setup_rabbit_cluster SSL is true",
"Wed Oct 30 15:15:44 UTC 2019 (debug) setup_rabbit_cluster CHECK_PORT is 5671",
"Wed Oct 30 15:15:44 UTC 2019 (debug) setup_rabbit_cluster First host in list is hegemone",
"Wed Oct 30 15:15:44 UTC 2019 (debug) setup_rabbit_cluster First host shortname is hegemone",
"Wed Oct 30 15:15:44 UTC 2019 (debug) setup_rabbit_cluster Primary Host is hegemone",
"Wed Oct 30 15:15:44 UTC 2019 (debug) setup_rabbit_cluster Primary Short Host is hegemone",
"Wed Oct 30 15:15:44 UTC 2019 (debug) setup_rabbit_cluster My Host is hegemone",
"Wed Oct 30 15:15:44 UTC 2019 (debug) setup_rabbit_cluster I am the primary host",
"Wed Oct 30 15:15:44 UTC 2019 setup_rabbit_cluster Copy generated Erlang shared secret to Rabbit.",
"Wed Oct 30 15:15:47 UTC 2019 setup_rabbit_cluster Copying generated Erlang shared secret to Rabbit.",
"Wed Oct 30 15:15:47 UTC 2019 setup_rabbit_cluster Starting sas-viya-rabbitmq-server-default",
"Wed Oct 30 15:16:07 UTC 2019 setup_rabbit_cluster The RabbitMQ service sas-viya-rabbitmq-server-default could not be started.",
"Wed Oct 30 15:16:07 UTC 2019 setup_rabbit_cluster Returned status was 1",
"Wed Oct 30 15:16:07 UTC 2019 setup_rabbit_cluster Setup failed. Beware of independent Rabbit hosts running with different shared secrets."
]
}

The deployment command was run using -vvv option for verbose debugging.

I have attempted to do a deploy cleanup, delete HTTPD, and reboot the server but I keep getting the same error.

Please find the log files from /opt/sas/viya/config/var/log/rabbitmq-server/default attached.

Any help is appreciated.

Thank you.

[alexal]

@DanBondarneko ,

RabbitMQ doesn't support short names. Each machine in the deployment must have a fully qualified domain name (FQDN). To ensure that each machine in the deployment has the host name that you expect, run the hostname, hostname -f, and the hostname -s commands on each machine. If any of the machines are not named as you expect or do not have an FQDN, correct the issue and run the commands again to confirm the correction.

For more information about the hostname command and its options, see the Linux man pages.

[DanBondarneko]

Hi,

Thank you for the reply. I corrected the issue with the FQDN and got past the error. However, now I get a different error:

failed: [deployTarget] (item={'_ansible_parsed': True, '_ansible_item_result': True, '_ansible_item_label': {u'SANMOUNT': u'/opt/sas/viya/config/data/sasdatasvrc', u'PCP_PORT': u'5430', u'PGPOOL_PORT': u'5431', u'SERVICE_NAME': u'postgres'}, u'ansible_job_id': u'672428512590.30664', 'failed': False, u'started': 1, 'changed': True, 'item': {u'SANMOUNT': u'/opt/sas/viya/config/data/sasdatasvrc', u'PCP_PORT': u'5430', u'PGPOOL_PORT': u'5431', u'SERVICE_NAME': u'postgres'}, u'finished': 0, u'results_file': u'/root/.ansible_async/672428512590.30664', '_ansible_ignore_errors': None, '_ansible_no_log': False}) => {"ansible_job_id": "672428512590.30664", "attempts": 2, "changed": true, "cmd": ["/opt/sas/viya/home/libexec/sasdatasvrc/script/sds_setup_pool.sh", "-config_path", "/opt/sas/viya/config/etc/sasdatasvrc/postgres/pgpool0/sds_env_var.sh"], "delta": "0:00:05.124353", "end": "2019-10-30 21:13:31.767888", "finished": 1, "item": {"ansible_job_id": "672428512590.30664", "changed": true, "failed": false, "finished": 0, "item": {"PCP_PORT": "5430", "PGPOOL_PORT": "5431", "SANMOUNT": "/opt/sas/viya/config/data/sasdatasvrc", "SERVICE_NAME": "postgres"}, "results_file": "/root/.ansible_async/672428512590.30664", "started": 1}, "msg": "non-zero return code", "rc": 1, "start": "2019-10-30 21:13:26.643535", "stderr": "level=error msg=\"Key already exists. Write failed.\" key=config/application/sas/database/postgres/username\nlevel=error msg=\"Key already exists. Write failed.\" key=config/application/sas/database/postgres/password\nError: Unable to perform the ssh key validation. Until the \"sas\" user can ssh from the \"localhost\" machine to the \"hegemone.clientspectrum.com\" machine, this setup will continue to fail.\n/opt/sas/viya/home/libexec/sasdatasvrc/script/sds_pool_config.sh ERROR: /opt/sas/viya/home/libexec/sasdatasvrc/script/sds_ssh_key_validation.sh \"hegemone.clientspectrum.com hegemone.clientspectrum.com\" failed. Terminating deployment...\nConfiguration of pool failed", "stderr_lines": ["level=error msg=\"Key already exists. Write failed.\" key=config/application/sas/database/postgres/username", "level=error msg=\"Key already exists. Write failed.\" key=config/application/sas/database/postgres/password", "Error: Unable to perform the ssh key validation. Until the \"sas\" user can ssh from the \"localhost\" machine to the \"hegemone.clientspectrum.com\" machine, this setup will continue to fail.", "/opt/sas/viya/home/libexec/sasdatasvrc/script/sds_pool_config.sh ERROR: /opt/sas/viya/home/libexec/sasdatasvrc/script/sds_ssh_key_validation.sh \"hegemone.clientspectrum.com hegemone.clientspectrum.com\" failed. Terminating deployment...", "Configuration of pool failed"], "stdout": "Started setup of SDS Postgres pool\nCalling sds_set_env_variable.sh...\n CONSUL_HTTP_ADDR=https://localhost:8501\n CONSUL_TEMPLATE_OPTIONS=-consul-addr localhost:8501 -consul-ssl -consul-ssl-ca-cert /opt/sas/viya/config/etc/SASSecurityCertificateFramework/cacerts/trustedcerts.pem\nmode of ‘/opt/sas/viya/config/etc/sasdatasvrc/postgres/pgpool0’ changed from 0755 (rwxr-xr-x) to 0700 (rwx------)\nmode of ‘/opt/sas/viya/config/data/sasdatasvrc/postgres/pgpool0’ changed from 0755 (rwxr-xr-x) to 0700 (rwx------)\nmode of ‘/opt/sas/viya/config/var/log/sasdatasvrc/postgres/pgpool0’ changed from 0755 (rwxr-xr-x) to 0750 (rwxr-x---)\n f_output_consul_short_service output:\n \"serviceAddress\": \"hegemone.clientspectrum.com\",\n \"serviceID\": \"postgres-datanode0\",\n \"servicePort\": 5432,\n \"primary\"\n\nCalling sds_pool_config.sh...", "stdout_lines": ["Started setup of SDS Postgres pool", "Calling sds_set_env_variable.sh...", " CONSUL_HTTP_ADDR=https://localhost:8501", " CONSUL_TEMPLATE_OPTIONS=-consul-addr localhost:8501 -consul-ssl -consul-ssl-ca-cert /opt/sas/viya/config/etc/SASSecurityCertificateFramework/cacerts/trustedcerts.pem", "mode of ‘/opt/sas/viya/config/etc/sasdatasvrc/postgres/pgpool0’ changed from 0755 (rwxr-xr-x) to 0700 (rwx------)", "mode of ‘/opt/sas/viya/config/data/sasdatasvrc/postgres/pgpool0’ changed from 0755 (rwxr-xr-x) to 0700 (rwx------)", "mode of ‘/opt/sas/viya/config/var/log/sasdatasvrc/postgres/pgpool0’ changed from 0755 (rwxr-xr-x) to 0750 (rwxr-x---)", " f_output_consul_short_service output:", " \"serviceAddress\": \"hegemone.clientspectrum.com\",", " \"serviceID\": \"postgres-datanode0\",", " \"servicePort\": 5432,", " \"primary\"", "", "Calling sds_pool_config.sh..."]}

Thank you.

[alexal]

@DanBondarneko ,

Error: Unable to perform the ssh key validation. Until the "sas" user can ssh from the "localhost" machine to the "hegemone.clientspectrum.com" machine, this setup will continue to fail.

Something is wrong with SSH keys for the "sas" user. Either SSH keys got broken or the "sas" user got locked.

[DanBondarneko]

Can you please elaborate what do you mean by SSH keys being broken? I created the sas user manually inside the sas group and I am doing a single-machine deployment. I can SSH to "hegemone.clientspectrum.com" using "ssh sas@hegemone.clientspectrum.com" and then entering the password.

Thank you

[alexal · Accepted solution]

@DanBondarneko ,

You need to create an SSH key for the "sas" user and add that key to an authorized_keys file. 

[DanBondarneko]

Thank you! There was an issue with the permissions for the keys file that caused problems. The deployment went smoothly after that. Thank you again.

[alexal]

@DanBondarneko ,

You're welcome! I'm glad that the problem has been resolved!