SAS viya 3.5 deployment issue

umamahesh — Tue, 16 May 2023 12:12:52 GMT

Hi Team,

while deploying sas software in two node getting below error, can any one please help me out

failed: [deployTarget] (item={'failed': 0, 'started': 1, 'finished': 0, 'ansible_job_id': '106300579898.334389', 'results_file': '/root/.ansible_async/106300579898.334389', 'changed': True, 'item': {'NODE_NUMBER': '0', 'PG_PORT': '5432', 'SANMOUNT': '/opt/sas/viya/config/data/sasdatasvrc', 'SERVICE_NAME': 'postgres'}, 'ansible_loop_var': 'item'}) => {

"ansible_job_id": "106300579898.334389",

"ansible_loop_var": "item",

"attempts": 55,

"changed": true,

"cmd": [

"bash",

"-c",

"/opt/sas/viya/home/libexec/sasdatasvrc/script/sds_setup_node.sh -config_path /opt/sas/viya/config/etc/sasdatasvrc/postgres/node0/sds_env_var.sh"

"delta": "0:09:11.630127",

"end": "2023-05-16 17:14:07.886968",

"finished": 1,

"invocation": {

"module_args": {

"_raw_params": "bash -c '/opt/sas/viya/home/libexec/sasdatasvrc/script/sds_setup_node.sh -config_path /opt/sas/viya/config/etc/sasdatasvrc/postgres/node0/sds_env_var.sh'",

"_uses_shell": false,

"argv": null,

"chdir": null,

"creates": null,

"executable": null,

"removes": null,

"stdin": null,

"stdin_add_newline": true,

"strip_empty_ends": true,

"warn": false

}

"item": {

"ansible_job_id": "106300579898.334389",

"ansible_loop_var": "item",

"changed": true,

"failed": 0,

"finished": 0,

"item": {

"NODE_NUMBER": "0",

"PG_PORT": "5432",

"SANMOUNT": "/opt/sas/viya/config/data/sasdatasvrc",

"SERVICE_NAME": "postgres"

"results_file": "/root/.ansible_async/106300579898.334389",

"started": 1

"msg": "non-zero return code",

"rc": 1,

"results_file": "/root/.ansible_async/106300579898.334389",

"start": "2023-05-16 17:04:56.256841",

"started": 1,

"stderr": "level=error app=sas-crypto-management timestamp=2023-05-16T11:35:03.166173839Z function=connection.(*vaultApiGetOperation).Perform msg=\"Failed to perform a GET API call against Vault\" endpoint=viya_inter/roles/test_web_server err=\"Get https://******.***.com:8200/v1/viya_inter/roles/test_web_server: Forbidden\"\nlevel=error app=sas-crypto-management timestamp=2023-05-16T11:35:07.702376417Z function=connection.(*vaultApiGetOperation).Perform msg=\"Failed to perform a GET API call against Vault\" endpoint=viya_inter/roles/test_web_server err=\"Get https://******.***.com:8200/v1/viya_inter/roles/test_web_server: Forbidden\"\nlevel=error app=sas-crypto-management timestamp=2023-05-16T11:35:13.612312424Z function=connection.(*vaultApiGetOperation).Perform msg=\"Failed to perform a GET API call against Vault\" endpoint=viya_inter/roles/test_web_server err=\"Get https://******.***.com:8200/v1/viya_inter/roles/test_web_server: Forbidden\"\nlevel=error app=sas-crypto-management timestamp=2023-05-16T11:35:21.427896974Z function=connection.(*vaultApiGetOperation).Perform msg=\"Failed to perform a GET API call against Vault\" endpoint=viya_inter/roles/test_web_server err=\"Get https://******.***.com:8200/v1/viya_inter/roles/test_web_server: Forbidden\"\nlevel=error app=sas-crypto-management timestamp=2023-05-16T11:35:33.484482245Z function=connection.(*vaultApiGetOperation).Perform msg=\"Failed to perform a GET API call against Vault\" endpoint=viya_inter/roles/test_web_server err=\"Get https://******.***.com:8200/v1/viya_inter/roles/test_web_server: Forbidden\"\nlevel=error app=sas-crypto-management timestamp=2023-05-16T11:35:52.991272444Z function=connection.(*vaultApiGetOperation).Perform msg=\"Failed to perform a GET API call against Vault\" endpoint=viya_inter/roles/test_web_server err=\"Get https://******.***.com:8200/v1/viya_inter/roles/test_web_server: Forbidden\"\nlevel=error app=sas-crypto-management timestamp=2023-05-16T11:36:29.026509233Z function=connection.(*vaultApiGetOperation).Perform msg=\"Failed to perform a GET API call against Vault\" endpoint=viya_inter/roles/test_web_server err=\"Get https://******.***.com:8200/v1/viya_inter/roles/test_web_server: Forbidden\"\nlevel=error app=sas-crypto-management timestamp=2023-05-16T11:37:36.510543909Z function=connection.(*vaultApiGetOperation).Perform msg=\"Failed to perform a GET API call against Vault\" endpoint=viya_inter/roles/test_web_server err=\"Get https://******.***.com:8200/v1/viya_inter/roles/test_web_server: Forbidden\"\nlevel=error app=sas-crypto-management timestamp=2023-05-16T11:39:47.884563448Z function=connection.(*vaultApiGetOperation).Perform msg=\"Failed to perform a GET API call against Vault\" endpoint=viya_inter/roles/test_web_server err=\"Get https://******.***.com:8200/v1/viya_inter/roles/test_web_server: Forbidden\"\nlevel=error app=sas-crypto-management timestamp=2023-05-16T11:44:07.848127688Z function=connection.(*vaultApiGetOperation).Perform msg=\"Failed to perform a GET API call against Vault\" endpoint=viya_inter/roles/test_web_server err=\"Get https://******.***.com:8200/v1/viya_inter/roles/test_web_server: Forbidden\"\nlevel=error app=sas-crypto-management timestamp=2023-05-16T11:44:07.8481765Z function=utils.(*TimeoutOperationPerformer).Perform msg=\"failed to perform the requested operation\" err=\"Get https://******.***.com:8200/v1/viya_inter/roles/test_web_server: Forbidden\"\nlevel=error app=sas-crypto-management timestamp=2023-05-16T11:44:07.848184895Z function=connection.(*VaultClient).getSecretFromPath msg=\"Failed to read vault secret path.\" path=viya_inter/roles/test_web_server err=\"Get https://******.***.com:8200/v1/viya_inter/roles/test_web_server: Forbidden\"\nlevel=error app=sas-crypto-management timestamp=2023-05-16T11:44:07.848193016Z function=connection.(*VaultClient).GetPrivateKeySettings msg=\"Failed to read private key settings\" endpoint=viya_inter/roles/test_web_server err=\"Get https://******.***.com:8200/v1/viya_inter/roles/test_web_server: Forbidden\"\nlevel=error app=sas-crypto-management timestamp=2023-05-16T11:44:07.848201851Z function=certificate.GenerateVaultCertificateWithCSR msg=\"failed to read private key configuration settings from Vault\" err=\"Get https://******.***.com:8200/v1/viya_inter/roles/test_web_server: Forbidden\"\nGet https://******.***.com:8200/v1/viya_inter/roles/test_web_server: Forbidden\nlevel=error app=sas-crypto-management timestamp=2023-05-16T11:44:07.848219979Z function=main.main err=\"Get https://******.***.com:8200/v1/viya_inter/roles/test_web_server: Forbidden\"\nERROR: f_generate_cert_key: failed '/opt/sas/viya/home/SASSecurityCertificateFramework/bin/sas-crypto-management certificate generate --certificate-generator vault --certificate-output-file /opt/sas/viya/config/etc/SASSecurityCertificateFramework/tls/certs/sasdatasvrc/postgres/node0/sascert.pem --private-key-output-file /opt/sas/viya/config/etc/SASSecurityCertificateFramework/private/sasdatasvrc/postgres/node0/saskey.pem --vault-service-token-input-file /opt/sas/viya/config/etc/SASSecurityCertificateFramework/tokens/sasdatasvrc/default/vault.token --trusted-ca-bundle /opt/sas/viya/config/etc/SASSecurityCertificateFramework/cacerts/trustedcerts.pem'. rC=1\nERROR: f_setup_operation_node_service: failed '/usr/sbin/runuser -l sas -c \"/bin/bash -c 'export CurrentLogFile=/opt/sas/viya/config/var/log/sasdatasvrc/postgres/node0/sds_setup_node.sh_20230516_170457_669.log; source /opt/sas/viya/home/libexec/sasdatasvrc/script/sds_set_env_variable.sh /opt/sas/viya/config/etc/sasdatasvrc/postgres/node0/sds_env_var.sh; f_generate_cert_key; exit $?'\"'. rC=1\nERROR: f_setup_node_main: failed 'f_setup_operation_node_service'. rC=1\nERROR: main: failed 'f_setup_node_main'. rC=1",

"stderr_lines": [

"level=error app=sas-crypto-management timestamp=2023-05-16T11:35:03.166173839Z function=connection.(*vaultApiGetOperation).Perform msg=\"Failed to perform a GET API call against Vault\" endpoint=viya_inter/roles/test_web_server err=\"Get https://******.***.com:8200/v1/viya_inter/roles/test_web_server: Forbidden\"",

"level=error app=sas-crypto-management timestamp=2023-05-16T11:35:07.702376417Z function=connection.(*vaultApiGetOperation).Perform msg=\"Failed to perform a GET API call against Vault\" endpoint=viya_inter/roles/test_web_server err=\"Get https://******.***.com:8200/v1/viya_inter/roles/test_web_server: Forbidden\"",

"level=error app=sas-crypto-management timestamp=2023-05-16T11:35:13.612312424Z function=connection.(*vaultApiGetOperation).Perform msg=\"Failed to perform a GET API call against Vault\" endpoint=viya_inter/roles/test_web_server err=\"Get https://******.***.com:8200/v1/viya_inter/roles/test_web_server: Forbidden\"",

"level=error app=sas-crypto-management timestamp=2023-05-16T11:35:21.427896974Z function=connection.(*vaultApiGetOperation).Perform msg=\"Failed to perform a GET API call against Vault\" endpoint=viya_inter/roles/test_web_server err=\"Get https://******.***.com:8200/v1/viya_inter/roles/test_web_server: Forbidden\"",

"level=error app=sas-crypto-management timestamp=2023-05-16T11:35:33.484482245Z function=connection.(*vaultApiGetOperation).Perform msg=\"Failed to perform a GET API call against Vault\" endpoint=viya_inter/roles/test_web_server err=\"Get https://******.***.com:8200/v1/viya_inter/roles/test_web_server: Forbidden\"",

"level=error app=sas-crypto-management timestamp=2023-05-16T11:35:52.991272444Z function=connection.(*vaultApiGetOperation).Perform msg=\"Failed to perform a GET API call against Vault\" endpoint=viya_inter/roles/test_web_server err=\"Get https://******.***.com:8200/v1/viya_inter/roles/test_web_server: Forbidden\"",

"level=error app=sas-crypto-management timestamp=2023-05-16T11:36:29.026509233Z function=connection.(*vaultApiGetOperation).Perform msg=\"Failed to perform a GET API call against Vault\" endpoint=viya_inter/roles/test_web_server err=\"Get https://******.***.com:8200/v1/viya_inter/roles/test_web_server: Forbidden\"",

"level=error app=sas-crypto-management timestamp=2023-05-16T11:37:36.510543909Z function=connection.(*vaultApiGetOperation).Perform msg=\"Failed to perform a GET API call against Vault\" endpoint=viya_inter/roles/test_web_server err=\"Get https://******.***.com:8200/v1/viya_inter/roles/test_web_server: Forbidden\"",

"level=error app=sas-crypto-management timestamp=2023-05-16T11:39:47.884563448Z function=connection.(*vaultApiGetOperation).Perform msg=\"Failed to perform a GET API call against Vault\" endpoint=viya_inter/roles/test_web_server err=\"Get https://******.***.com:8200/v1/viya_inter/roles/test_web_server: Forbidden\"",

"level=error app=sas-crypto-management timestamp=2023-05-16T11:44:07.848127688Z function=connection.(*vaultApiGetOperation).Perform msg=\"Failed to perform a GET API call against Vault\" endpoint=viya_inter/roles/test_web_server err=\"Get https://******.***.com:8200/v1/viya_inter/roles/test_web_server: Forbidden\"",

"level=error app=sas-crypto-management timestamp=2023-05-16T11:44:07.8481765Z function=utils.(*TimeoutOperationPerformer).Perform msg=\"failed to perform the requested operation\" err=\"Get https://******.***.com:8200/v1/viya_inter/roles/test_web_server: Forbidden\"",

"level=error app=sas-crypto-management timestamp=2023-05-16T11:44:07.848184895Z function=connection.(*VaultClient).getSecretFromPath msg=\"Failed to read vault secret path.\" path=viya_inter/roles/test_web_server err=\"Get https://******.***.com:8200/v1/viya_inter/roles/test_web_server: Forbidden\"",

"level=error app=sas-crypto-management timestamp=2023-05-16T11:44:07.848193016Z function=connection.(*VaultClient).GetPrivateKeySettings msg=\"Failed to read private key settings\" endpoint=viya_inter/roles/test_web_server err=\"Get https://******.***.com:8200/v1/viya_inter/roles/test_web_server: Forbidden\"",

"level=error app=sas-crypto-management timestamp=2023-05-16T11:44:07.848201851Z function=certificate.GenerateVaultCertificateWithCSR msg=\"failed to read private key configuration settings from Vault\" err=\"Get https://******.***.com:8200/v1/viya_inter/roles/test_web_server: Forbidden\"",

"Get https://******.***.com:8200/v1/viya_inter/roles/test_web_server: Forbidden",

"level=error app=sas-crypto-management timestamp=2023-05-16T11:44:07.848219979Z function=main.main err=\"Get https://******.***.com:8200/v1/viya_inter/roles/test_web_server: Forbidden\"",

"ERROR: f_generate_cert_key: failed '/opt/sas/viya/home/SASSecurityCertificateFramework/bin/sas-crypto-management certificate generate --certificate-generator vault --certificate-output-file /opt/sas/viya/config/etc/SASSecurityCertificateFramework/tls/certs/sasdatasvrc/postgres/node0/sascert.pem --private-key-output-file /opt/sas/viya/config/etc/SASSecurityCertificateFramework/private/sasdatasvrc/postgres/node0/saskey.pem --vault-service-token-input-file /opt/sas/viya/config/etc/SASSecurityCertificateFramework/tokens/sasdatasvrc/default/vault.token --trusted-ca-bundle /opt/sas/viya/config/etc/SASSecurityCertificateFramework/cacerts/trustedcerts.pem'. rC=1",

"ERROR: f_setup_operation_node_service: failed '/usr/sbin/runuser -l sas -c \"/bin/bash -c 'export CurrentLogFile=/opt/sas/viya/config/var/log/sasdatasvrc/postgres/node0/sds_setup_node.sh_20230516_170457_669.log; source /opt/sas/viya/home/libexec/sasdatasvrc/script/sds_set_env_variable.sh /opt/sas/viya/config/etc/sasdatasvrc/postgres/node0/sds_env_var.sh; f_generate_cert_key; exit $?'\"'. rC=1",

"ERROR: f_setup_node_main: failed 'f_setup_operation_node_service'. rC=1",

"ERROR: main: failed 'f_setup_node_main'. rC=1"

"stdout": "source /opt/sas/viya/home/libexec/sasdatasvrc/script/sds_set_env_variable.sh /opt/sas/viya/config/etc/sasdatasvrc/postgres/node0/sds_env_var.sh\n/opt/sas/viya/home/libexec/sasdatasvrc/script/sds_setup_node.sh sourcing sds_set_env_variable with /opt/sas/viya/config/etc/sasdatasvrc/postgres/node0/sds_env_var.sh...\n SAS_HOSTNAME=******.***.com, SAS_SERVICE_ADDR=******.***.com, SAS_BIND_ADDR=0.0.0.0\n VAULT_ADDR: https://******.***.com:8200\n SASCONFIG=/opt/sas/viya/config, PGHOME=/opt/sas/viya/home/postgresql11, PGPOOLHOME=/opt/sas/viya/home/pgpool-II40\n POSTGRES_FULL_VERSION=11.17, PGPOOL_FULL_VERSION=4.0.17\nSourced. HA_PGPOOL_COUNT=0, SASCONFIG=/opt/sas/viya/config, SAS_LOG_DIR=/opt/sas/viya/config/var/log",

"stdout_lines": [

"source /opt/sas/viya/home/libexec/sasdatasvrc/script/sds_set_env_variable.sh /opt/sas/viya/config/etc/sasdatasvrc/postgres/node0/sds_env_var.sh",

"/opt/sas/viya/home/libexec/sasdatasvrc/script/sds_setup_node.sh sourcing sds_set_env_variable with /opt/sas/viya/config/etc/sasdatasvrc/postgres/node0/sds_env_var.sh...",

" SAS_HOSTNAME=******.***.com, SAS_SERVICE_ADDR=******.***.com, SAS_BIND_ADDR=0.0.0.0",

" VAULT_ADDR: https://******.***.com:8200",

" SASCONFIG=/opt/sas/viya/config, PGHOME=/opt/sas/viya/home/postgresql11, PGPOOLHOME=/opt/sas/viya/home/pgpool-II40",

" POSTGRES_FULL_VERSION=11.17, PGPOOL_FULL_VERSION=4.0.17",

"Sourced. HA_PGPOOL_COUNT=0, SASCONFIG=/opt/sas/viya/config, SAS_LOG_DIR=/opt/sas/viya/config/var/log"

]

why its showing forbidden erro, where should i check permissions.

Regards,

Umamahesh.

Re: SAS viya 3.5 deployment issue

gwootton — Tue, 16 May 2023 13:46:00 GMT

This error is a failure engaging with the SAS Secrets Manager (vault) to generate a certificate for postgres, so you may want to review the logs for vault (/opt/sas/viya/config/var/log/vault/default)
https://go.documentation.sas.com/doc/en/calcdc/3.5/calsrvinf/n00002viyainfrsrvs00000admin.htm

Re: SAS viya 3.5 deployment issue

rushabh_45 — Thu, 01 Jun 2023 05:23:46 GMT

Hi George,

I am also facing the same issue. Can you please provide your help on this

Re: SAS viya 3.5 deployment issue

rushabh_45 — Thu, 01 Jun 2023 05:36:58 GMT

*Greg

Re: SAS viya 3.5 deployment issue

gwootton — Thu, 01 Jun 2023 13:49:09 GMT

If you are encountering the same failure then my advice would be the same, to check the logs for SAS Secrets Manager for any issues.

Re: SAS viya 3.5 deployment issue

MargaretC — Thu, 01 Jun 2023 13:53:41 GMT

What version of Linux are you installing Viya 3.5 on?

topic Re: SAS viya 3.5 deployment issue in Administration and Deployment

SAS viya 3.5 deployment issue

Re: SAS viya 3.5 deployment issue

Re: SAS viya 3.5 deployment issue

Re: SAS viya 3.5 deployment issue

Re: SAS viya 3.5 deployment issue

Re: SAS viya 3.5 deployment issue