SAS Viya 3.4 multiple machines intermediate cert Error from Vault server

[Asif4862]

Hello All, This is Asif

I deployed SAS Viya 3.4 on multiple machines (Install MPP CAS on 2 hosts)

Host1= CAS Controller and all services
Host2= CAS worker

During installation of SAS Viya 3.4 only one error is occurred on Host2 (cas-worker-1):

"2020-08-10 12:19:08,892 p=10363 u=root |  fatal: [cas-worker-1]: FAILED! => {"changed": true, "cmd": "/opt/sas/viya/home/SASSecurityCertificateFramework/bin/sas-crypto-management req-vault-cert --common-name host1.local --out-form jks --out-key /opt/sas/viya/config/etc/SASSecurityCertificateFramework/private/sas.jks --san-ip 127.0.0.1 --san-ip 172.31.0.13 --vault-addr https://host1.local:8200 --vault-token /opt/sas/viya/config/etc/SASSecurityCertificateFramework/tokens/staticcerts/default/vault.token --vault-cafile /opt/sas/viya/config/etc/SASSecurityCertificateFramework/cacerts/trustedcerts.pem", "delta": "0:13:31.142453", "end": "2020-08-10 12:19:08.845065", "msg": "non-zero return code", "rc": 1, "start": "2020-08-10 12:05:37.702612", "stderr": "2020-08-10 12:06:07.828 INFO [Log.go:41] [sas-crypto-management-command] - {\"0\":\"[]\"} vaultcert.go:423 Will retry, but failed to get intermediate cert from Vault server.\n2020-08-10 12:06:38.831 INFO [Log.go:41] [sas-crypto-management-command] - {\"0\":\"[]\"} vaultcert.go:423 Will retry, but failed to get intermediate cert from Vault server.\n2020-08-10 12:07:10.832 INFO [Log.go:41] [sas-crypto-management-command] - {\"0\":\"[]\"} vaultcert.go:423 Will retry, but failed to get intermediate cert from Vault server.\n2020-08-10 12:07:44.834 INFO [Log.go:41] [sas-crypto-management-command] - {\"0\":\"[]\"} vaultcert.go:423 Will retry, but failed to get intermediate cert from Vault server.\n2020-08-10 12:08:22.835 INFO [Log.go:41] [sas-crypto-management-command] - {\"0\":\"[]\"} vaultcert.go:423 Will retry, but failed to get intermediate cert from Vault server.\n2020-08-10 12:09:08.837 INFO [Log.go:41] [sas-crypto-management-command] - {\"0\":\"[]\"} vaultcert.go:423 Will retry, but failed to get intermediate cert from Vault server.\n2020-08-10 12:10:10.838 INFO [Log.go:41] [sas-crypto-management-command] - {\"0\":\"[]\"} vaultcert.go:423 Will retry, but failed to get intermediate cert from Vault server.\n2020-08-10 12:11:44.840 INFO [Log.go:41] [sas-crypto-management-command] - {\"0\":\"[]\"} vaultcert.go:423 Will retry, but failed to get intermediate cert from Vault server.\n2020-08-10 12:14:22.841 INFO [Log.go:41] [sas-crypto-management-command] - {\"0\":\"[]\"} vaultcert.go:423 Will retry, but failed to get intermediate cert from Vault server.\n2020-08-10 12:19:08.843 ERROR [Log.go:57] [sas-crypto-management-command] - {\"0\":\"[]\"} vaultcert.go:99 After 511 seconds, failed to read viya_inter/ca/pem: Get https://host1.local:8200/v1/viya_inter/ca/pem: dial tcp 172.1.1.1:8200: i/o timeout", "stderr_lines": ["2020-08-10 12:06:07.828 INFO [Log.go:41] [sas-crypto-management-command] - {\"0\":\"[]\"} vaultcert.go:423 Will retry, but failed to get intermediate cert from Vault server.", "2020-08-10 12:06:38.831 INFO [Log.go:41] [sas-crypto-management-command] - {\"0\":\"[]\"} vaultcert.go:423 Will retry, but failed to get intermediate cert from Vault server.", "2020-08-10 12:07:10.832 INFO [Log.go:41] [sas-crypto-management-command] - {\"0\":\"[]\"} vaultcert.go:423 Will retry, but failed to get intermediate cert from Vault server.", "2020-08-10 12:07:44.834 INFO [Log.go:41] [sas-crypto-management-command] - {\"0\":\"[]\"} vaultcert.go:423 Will retry, but failed to get intermediate cert from Vault server.", "2020-08-10 12:08:22.835 INFO [Log.go:41] [sas-crypto-management-command] - {\"0\":\"[]\"} vaultcert.go:423 Will retry, but failed to get intermediate cert from Vault server.", "2020-08-10 12:09:08.837 INFO [Log.go:41] [sas-crypto-management-command] - {\"0\":\"[]\"} vaultcert.go:423 Will retry, but failed to get intermediate cert from Vault server.", "2020-08-10 12:10:10.838 INFO [Log.go:41] [sas-crypto-management-command] - {\"0\":\"[]\"} vaultcert.go:423 Will retry, but failed to get intermediate cert from Vault server.", "2020-08-10 12:11:44.840 INFO [Log.go:41] [sas-crypto-management-command] - {\"0\":\"[]\"} vaultcert.go:423 Will retry, but failed to get intermediate cert from Vault server.", "2020-08-10 12:14:22.841 INFO [Log.go:41] [sas-crypto-management-command] - {\"0\":\"[]\"} vaultcert.go:423 Will retry, but failed to get intermediate cert from Vault server.", "2020-08-10 12:19:08.843 ERROR [Log.go:57] [sas-crypto-management-command] - {\"0\":\"[]\"} vaultcert.go:99 After 511 seconds, failed to read viya_inter/ca/pem: Get https://host1.local:8200/v1/viya_inter/ca/pem: dial tcp 172.1.1.1:8200: i/o timeout"], "stdout": "", "stdout_lines": []}"

2020-08-10 12:19:08,893 p=10363 u=root |  NO MORE HOSTS LEFT *************************************************************

2020-08-10 12:19:08,893 p=10363 u=root |  PLAY RECAP *********************************************************************

2020-08-10 12:19:08,893 p=10363 u=root |  cas-worker-1               : ok=143  changed=32   unreachable=0    failed=1  

2020-08-10 12:19:08,894 p=10363 u=root |  deployTarget               : ok=261  changed=97   unreachable=0    failed=0  

2020-08-10 12:19:08,894 p=10363 u=root |  localhost                  : ok=14   changed=1    unreachable=0    failed=0  

Do you have any suggestion or recommendation how to resolve that specific error?

Thanks in advance

Thanks,
Asif

[alexal]

@Asif4862 .

Do you have a firewall between those machines? Please run the following command on cas-worker-1 and show the output:

curl -v -k https://host1.local:8200/v1/viya_inter/ca/pem

[Asif4862 · Accepted solution]

@alexal , actually you are right. There are firewall between two servers. 

The firewall permission issue was related to the Security Group attached to the EC2 instances. The two instances are in different subnets. The Security Group attached to first instance did not have an inbound entry for traffic from the second instances subnet.

I basically added an inbound rule to the first instances Security Group to allow traffic from second instances subnet.

The second instances Security Group also allows traffic from first instances subnet.

Once again thank you for your help

Thanks,
Asif

[alexal]

@Asif4862 ,

You're welcome. I'm glad the problem has been resolved.

[Asif4862]

Thanks @alexal  and appreciated 🙂