BookmarkSubscribeRSS Feed
umamahesh
Calcite | Level 5

Hi Team,

 

while deploying sas software in two node getting below error, can any one please help me out

 

failed: [deployTarget] (item={'failed': 0, 'started': 1, 'finished': 0, 'ansible_job_id': '106300579898.334389', 'results_file': '/root/.ansible_async/106300579898.334389', 'changed': True, 'item': {'NODE_NUMBER': '0', 'PG_PORT': '5432', 'SANMOUNT': '/opt/sas/viya/config/data/sasdatasvrc', 'SERVICE_NAME': 'postgres'}, 'ansible_loop_var': 'item'}) => {

    "ansible_job_id": "106300579898.334389",

    "ansible_loop_var": "item",

    "attempts": 55,

    "changed": true,

    "cmd": [

        "bash",

        "-c",

        "/opt/sas/viya/home/libexec/sasdatasvrc/script/sds_setup_node.sh -config_path /opt/sas/viya/config/etc/sasdatasvrc/postgres/node0/sds_env_var.sh"

    ],

    "delta": "0:09:11.630127",

    "end": "2023-05-16 17:14:07.886968",

    "finished": 1,

    "invocation": {

        "module_args": {

            "_raw_params": "bash -c '/opt/sas/viya/home/libexec/sasdatasvrc/script/sds_setup_node.sh -config_path /opt/sas/viya/config/etc/sasdatasvrc/postgres/node0/sds_env_var.sh'",

            "_uses_shell": false,

            "argv": null,

            "chdir": null,

            "creates": null,

            "executable": null,

            "removes": null,

            "stdin": null,

            "stdin_add_newline": true,

            "strip_empty_ends": true,

            "warn": false

        }

    },

    "item": {

        "ansible_job_id": "106300579898.334389",

        "ansible_loop_var": "item",

        "changed": true,

        "failed": 0,

        "finished": 0,

        "item": {

            "NODE_NUMBER": "0",

            "PG_PORT": "5432",

            "SANMOUNT": "/opt/sas/viya/config/data/sasdatasvrc",

            "SERVICE_NAME": "postgres"

        },

        "results_file": "/root/.ansible_async/106300579898.334389",

        "started": 1

    },

    "msg": "non-zero return code",

    "rc": 1,

    "results_file": "/root/.ansible_async/106300579898.334389",

    "start": "2023-05-16 17:04:56.256841",

    "started": 1,

    "stderr": "level=error app=sas-crypto-management timestamp=2023-05-16T11:35:03.166173839Z function=connection.(*vaultApiGetOperation).Perform msg=\"Failed to perform a GET API call against Vault\" endpoint=viya_inter/roles/test_web_server err=\"Get https://******.***.com:8200/v1/viya_inter/roles/test_web_server: Forbidden\"\nlevel=error app=sas-crypto-management timestamp=2023-05-16T11:35:07.702376417Z function=connection.(*vaultApiGetOperation).Perform msg=\"Failed to perform a GET API call against Vault\" endpoint=viya_inter/roles/test_web_server err=\"Get https://******.***.com:8200/v1/viya_inter/roles/test_web_server: Forbidden\"\nlevel=error app=sas-crypto-management timestamp=2023-05-16T11:35:13.612312424Z function=connection.(*vaultApiGetOperation).Perform msg=\"Failed to perform a GET API call against Vault\" endpoint=viya_inter/roles/test_web_server err=\"Get https://******.***.com:8200/v1/viya_inter/roles/test_web_server: Forbidden\"\nlevel=error app=sas-crypto-management timestamp=2023-05-16T11:35:21.427896974Z function=connection.(*vaultApiGetOperation).Perform msg=\"Failed to perform a GET API call against Vault\" endpoint=viya_inter/roles/test_web_server err=\"Get https://******.***.com:8200/v1/viya_inter/roles/test_web_server: Forbidden\"\nlevel=error app=sas-crypto-management timestamp=2023-05-16T11:35:33.484482245Z function=connection.(*vaultApiGetOperation).Perform msg=\"Failed to perform a GET API call against Vault\" endpoint=viya_inter/roles/test_web_server err=\"Get https://******.***.com:8200/v1/viya_inter/roles/test_web_server: Forbidden\"\nlevel=error app=sas-crypto-management timestamp=2023-05-16T11:35:52.991272444Z function=connection.(*vaultApiGetOperation).Perform msg=\"Failed to perform a GET API call against Vault\" endpoint=viya_inter/roles/test_web_server err=\"Get https://******.***.com:8200/v1/viya_inter/roles/test_web_server: Forbidden\"\nlevel=error app=sas-crypto-management timestamp=2023-05-16T11:36:29.026509233Z function=connection.(*vaultApiGetOperation).Perform msg=\"Failed to perform a GET API call against Vault\" endpoint=viya_inter/roles/test_web_server err=\"Get https://******.***.com:8200/v1/viya_inter/roles/test_web_server: Forbidden\"\nlevel=error app=sas-crypto-management timestamp=2023-05-16T11:37:36.510543909Z function=connection.(*vaultApiGetOperation).Perform msg=\"Failed to perform a GET API call against Vault\" endpoint=viya_inter/roles/test_web_server err=\"Get https://******.***.com:8200/v1/viya_inter/roles/test_web_server: Forbidden\"\nlevel=error app=sas-crypto-management timestamp=2023-05-16T11:39:47.884563448Z function=connection.(*vaultApiGetOperation).Perform msg=\"Failed to perform a GET API call against Vault\" endpoint=viya_inter/roles/test_web_server err=\"Get https://******.***.com:8200/v1/viya_inter/roles/test_web_server: Forbidden\"\nlevel=error app=sas-crypto-management timestamp=2023-05-16T11:44:07.848127688Z function=connection.(*vaultApiGetOperation).Perform msg=\"Failed to perform a GET API call against Vault\" endpoint=viya_inter/roles/test_web_server err=\"Get https://******.***.com:8200/v1/viya_inter/roles/test_web_server: Forbidden\"\nlevel=error app=sas-crypto-management timestamp=2023-05-16T11:44:07.8481765Z function=utils.(*TimeoutOperationPerformer).Perform msg=\"failed to perform the requested operation\" err=\"Get https://******.***.com:8200/v1/viya_inter/roles/test_web_server: Forbidden\"\nlevel=error app=sas-crypto-management timestamp=2023-05-16T11:44:07.848184895Z function=connection.(*VaultClient).getSecretFromPath msg=\"Failed to read vault secret path.\" path=viya_inter/roles/test_web_server err=\"Get https://******.***.com:8200/v1/viya_inter/roles/test_web_server: Forbidden\"\nlevel=error app=sas-crypto-management timestamp=2023-05-16T11:44:07.848193016Z function=connection.(*VaultClient).GetPrivateKeySettings msg=\"Failed to read private key settings\" endpoint=viya_inter/roles/test_web_server err=\"Get https://******.***.com:8200/v1/viya_inter/roles/test_web_server: Forbidden\"\nlevel=error app=sas-crypto-management timestamp=2023-05-16T11:44:07.848201851Z function=certificate.GenerateVaultCertificateWithCSR msg=\"failed to read private key configuration settings from Vault\" err=\"Get https://******.***.com:8200/v1/viya_inter/roles/test_web_server: Forbidden\"\nGet https://******.***.com:8200/v1/viya_inter/roles/test_web_server: Forbidden\nlevel=error app=sas-crypto-management timestamp=2023-05-16T11:44:07.848219979Z function=main.main err=\"Get https://******.***.com:8200/v1/viya_inter/roles/test_web_server: Forbidden\"\nERROR: f_generate_cert_key: failed '/opt/sas/viya/home/SASSecurityCertificateFramework/bin/sas-crypto-management certificate generate --certificate-generator vault --certificate-output-file /opt/sas/viya/config/etc/SASSecurityCertificateFramework/tls/certs/sasdatasvrc/postgres/node0/sascert.pem --private-key-output-file /opt/sas/viya/config/etc/SASSecurityCertificateFramework/private/sasdatasvrc/postgres/node0/saskey.pem --vault-service-token-input-file /opt/sas/viya/config/etc/SASSecurityCertificateFramework/tokens/sasdatasvrc/default/vault.token --trusted-ca-bundle /opt/sas/viya/config/etc/SASSecurityCertificateFramework/cacerts/trustedcerts.pem'. rC=1\nERROR: f_setup_operation_node_service: failed '/usr/sbin/runuser -l sas -c \"/bin/bash -c 'export CurrentLogFile=/opt/sas/viya/config/var/log/sasdatasvrc/postgres/node0/sds_setup_node.sh_20230516_170457_669.log; source /opt/sas/viya/home/libexec/sasdatasvrc/script/sds_set_env_variable.sh /opt/sas/viya/config/etc/sasdatasvrc/postgres/node0/sds_env_var.sh; f_generate_cert_key; exit $?'\"'. rC=1\nERROR: f_setup_node_main: failed 'f_setup_operation_node_service'. rC=1\nERROR: main: failed 'f_setup_node_main'. rC=1",

    "stderr_lines": [

        "level=error app=sas-crypto-management timestamp=2023-05-16T11:35:03.166173839Z function=connection.(*vaultApiGetOperation).Perform msg=\"Failed to perform a GET API call against Vault\" endpoint=viya_inter/roles/test_web_server err=\"Get https://******.***.com:8200/v1/viya_inter/roles/test_web_server: Forbidden\"",

        "level=error app=sas-crypto-management timestamp=2023-05-16T11:35:07.702376417Z function=connection.(*vaultApiGetOperation).Perform msg=\"Failed to perform a GET API call against Vault\" endpoint=viya_inter/roles/test_web_server err=\"Get https://******.***.com:8200/v1/viya_inter/roles/test_web_server: Forbidden\"",

        "level=error app=sas-crypto-management timestamp=2023-05-16T11:35:13.612312424Z function=connection.(*vaultApiGetOperation).Perform msg=\"Failed to perform a GET API call against Vault\" endpoint=viya_inter/roles/test_web_server err=\"Get https://******.***.com:8200/v1/viya_inter/roles/test_web_server: Forbidden\"",

        "level=error app=sas-crypto-management timestamp=2023-05-16T11:35:21.427896974Z function=connection.(*vaultApiGetOperation).Perform msg=\"Failed to perform a GET API call against Vault\" endpoint=viya_inter/roles/test_web_server err=\"Get https://******.***.com:8200/v1/viya_inter/roles/test_web_server: Forbidden\"",

        "level=error app=sas-crypto-management timestamp=2023-05-16T11:35:33.484482245Z function=connection.(*vaultApiGetOperation).Perform msg=\"Failed to perform a GET API call against Vault\" endpoint=viya_inter/roles/test_web_server err=\"Get https://******.***.com:8200/v1/viya_inter/roles/test_web_server: Forbidden\"",

        "level=error app=sas-crypto-management timestamp=2023-05-16T11:35:52.991272444Z function=connection.(*vaultApiGetOperation).Perform msg=\"Failed to perform a GET API call against Vault\" endpoint=viya_inter/roles/test_web_server err=\"Get https://******.***.com:8200/v1/viya_inter/roles/test_web_server: Forbidden\"",

        "level=error app=sas-crypto-management timestamp=2023-05-16T11:36:29.026509233Z function=connection.(*vaultApiGetOperation).Perform msg=\"Failed to perform a GET API call against Vault\" endpoint=viya_inter/roles/test_web_server err=\"Get https://******.***.com:8200/v1/viya_inter/roles/test_web_server: Forbidden\"",

        "level=error app=sas-crypto-management timestamp=2023-05-16T11:37:36.510543909Z function=connection.(*vaultApiGetOperation).Perform msg=\"Failed to perform a GET API call against Vault\" endpoint=viya_inter/roles/test_web_server err=\"Get https://******.***.com:8200/v1/viya_inter/roles/test_web_server: Forbidden\"",

        "level=error app=sas-crypto-management timestamp=2023-05-16T11:39:47.884563448Z function=connection.(*vaultApiGetOperation).Perform msg=\"Failed to perform a GET API call against Vault\" endpoint=viya_inter/roles/test_web_server err=\"Get https://******.***.com:8200/v1/viya_inter/roles/test_web_server: Forbidden\"",

        "level=error app=sas-crypto-management timestamp=2023-05-16T11:44:07.848127688Z function=connection.(*vaultApiGetOperation).Perform msg=\"Failed to perform a GET API call against Vault\" endpoint=viya_inter/roles/test_web_server err=\"Get https://******.***.com:8200/v1/viya_inter/roles/test_web_server: Forbidden\"",

        "level=error app=sas-crypto-management timestamp=2023-05-16T11:44:07.8481765Z function=utils.(*TimeoutOperationPerformer).Perform msg=\"failed to perform the requested operation\" err=\"Get https://******.***.com:8200/v1/viya_inter/roles/test_web_server: Forbidden\"",

        "level=error app=sas-crypto-management timestamp=2023-05-16T11:44:07.848184895Z function=connection.(*VaultClient).getSecretFromPath msg=\"Failed to read vault secret path.\" path=viya_inter/roles/test_web_server err=\"Get https://******.***.com:8200/v1/viya_inter/roles/test_web_server: Forbidden\"",

        "level=error app=sas-crypto-management timestamp=2023-05-16T11:44:07.848193016Z function=connection.(*VaultClient).GetPrivateKeySettings msg=\"Failed to read private key settings\" endpoint=viya_inter/roles/test_web_server err=\"Get https://******.***.com:8200/v1/viya_inter/roles/test_web_server: Forbidden\"",

        "level=error app=sas-crypto-management timestamp=2023-05-16T11:44:07.848201851Z function=certificate.GenerateVaultCertificateWithCSR msg=\"failed to read private key configuration settings from Vault\" err=\"Get https://******.***.com:8200/v1/viya_inter/roles/test_web_server: Forbidden\"",

        "Get https://******.***.com:8200/v1/viya_inter/roles/test_web_server: Forbidden",

        "level=error app=sas-crypto-management timestamp=2023-05-16T11:44:07.848219979Z function=main.main err=\"Get https://******.***.com:8200/v1/viya_inter/roles/test_web_server: Forbidden\"",

        "ERROR: f_generate_cert_key: failed '/opt/sas/viya/home/SASSecurityCertificateFramework/bin/sas-crypto-management certificate generate --certificate-generator vault --certificate-output-file /opt/sas/viya/config/etc/SASSecurityCertificateFramework/tls/certs/sasdatasvrc/postgres/node0/sascert.pem --private-key-output-file /opt/sas/viya/config/etc/SASSecurityCertificateFramework/private/sasdatasvrc/postgres/node0/saskey.pem --vault-service-token-input-file /opt/sas/viya/config/etc/SASSecurityCertificateFramework/tokens/sasdatasvrc/default/vault.token --trusted-ca-bundle /opt/sas/viya/config/etc/SASSecurityCertificateFramework/cacerts/trustedcerts.pem'. rC=1",

        "ERROR: f_setup_operation_node_service: failed '/usr/sbin/runuser -l sas -c \"/bin/bash -c 'export CurrentLogFile=/opt/sas/viya/config/var/log/sasdatasvrc/postgres/node0/sds_setup_node.sh_20230516_170457_669.log; source /opt/sas/viya/home/libexec/sasdatasvrc/script/sds_set_env_variable.sh /opt/sas/viya/config/etc/sasdatasvrc/postgres/node0/sds_env_var.sh; f_generate_cert_key; exit $?'\"'. rC=1",

        "ERROR: f_setup_node_main: failed 'f_setup_operation_node_service'. rC=1",

        "ERROR: main: failed 'f_setup_node_main'. rC=1"

    ],

    "stdout": "source /opt/sas/viya/home/libexec/sasdatasvrc/script/sds_set_env_variable.sh /opt/sas/viya/config/etc/sasdatasvrc/postgres/node0/sds_env_var.sh\n/opt/sas/viya/home/libexec/sasdatasvrc/script/sds_setup_node.sh sourcing sds_set_env_variable with /opt/sas/viya/config/etc/sasdatasvrc/postgres/node0/sds_env_var.sh...\n  SAS_HOSTNAME=******.***.com, SAS_SERVICE_ADDR=******.***.com, SAS_BIND_ADDR=0.0.0.0\n  VAULT_ADDR: https://******.***.com:8200\n  SASCONFIG=/opt/sas/viya/config, PGHOME=/opt/sas/viya/home/postgresql11, PGPOOLHOME=/opt/sas/viya/home/pgpool-II40\n  POSTGRES_FULL_VERSION=11.17, PGPOOL_FULL_VERSION=4.0.17\nSourced. HA_PGPOOL_COUNT=0, SASCONFIG=/opt/sas/viya/config, SAS_LOG_DIR=/opt/sas/viya/config/var/log",

    "stdout_lines": [

        "source /opt/sas/viya/home/libexec/sasdatasvrc/script/sds_set_env_variable.sh /opt/sas/viya/config/etc/sasdatasvrc/postgres/node0/sds_env_var.sh",

        "/opt/sas/viya/home/libexec/sasdatasvrc/script/sds_setup_node.sh sourcing sds_set_env_variable with /opt/sas/viya/config/etc/sasdatasvrc/postgres/node0/sds_env_var.sh...",

        "  SAS_HOSTNAME=******.***.com, SAS_SERVICE_ADDR=******.***.com, SAS_BIND_ADDR=0.0.0.0",

        "  VAULT_ADDR: https://******.***.com:8200",

        "  SASCONFIG=/opt/sas/viya/config, PGHOME=/opt/sas/viya/home/postgresql11, PGPOOLHOME=/opt/sas/viya/home/pgpool-II40",

        "  POSTGRES_FULL_VERSION=11.17, PGPOOL_FULL_VERSION=4.0.17",

        "Sourced. HA_PGPOOL_COUNT=0, SASCONFIG=/opt/sas/viya/config, SAS_LOG_DIR=/opt/sas/viya/config/var/log"

    ]

 

 

 

why its showing forbidden erro, where should i check permissions.

 

Regards,

Umamahesh.

5 REPLIES 5
gwootton
SAS Super FREQ
This error is a failure engaging with the SAS Secrets Manager (vault) to generate a certificate for postgres, so you may want to review the logs for vault (/opt/sas/viya/config/var/log/vault/default)
https://go.documentation.sas.com/doc/en/calcdc/3.5/calsrvinf/n00002viyainfrsrvs00000admin.htm
--
Greg Wootton | Principal Systems Technical Support Engineer
rushabh_45
Calcite | Level 5

Hi George,

I am also facing the same issue. Can you please provide your help on this

rushabh_45
Calcite | Level 5

*Greg

gwootton
SAS Super FREQ
If you are encountering the same failure then my advice would be the same, to check the logs for SAS Secrets Manager for any issues.
--
Greg Wootton | Principal Systems Technical Support Engineer
MargaretC
SAS Employee

What version of Linux are you installing Viya 3.5 on?

suga badge.PNGThe SAS Users Group for Administrators (SUGA) is open to all SAS administrators and architects who install, update, manage or maintain a SAS deployment. 

Join SUGA 

Get Started with SAS Information Catalog in SAS Viya

SAS technical trainer Erin Winters shows you how to explore assets, create new data discovery agents, schedule data discovery agents, and much more.

Find more tutorials on the SAS Users YouTube channel.

Discussion stats
  • 5 replies
  • 1276 views
  • 0 likes
  • 4 in conversation