cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
BookmarkSubscribeRSS Feed

SAS Viya 2020.1 (and later) TLS Troubleshooting Tips

Started ‎12-15-2020 by
Modified ‎12-15-2020 by
Views 6,217

Continuing the series looking at TLS with SAS Viya 2020.1(and later), this time we will look at some troubleshooting tips. We will use Kubernetes client commands to validate the expected items are created and in a ready state. Also, we’ll look at the logging for the sas-certframe init container and see how the different steps we expect it to complete are reflected in the log messages.

 

Since the default configuration for SAS Viya 2020.1(and later) is full-stack TLS this is the environment we will use for these troubleshooting tips. Specifically, we will look at an environment where the customer has provided a certificate pair (private key and signed server identity certificate) for the Ingress Controller. All other SAS services will be using the SAS generated cert-manager issuer. This is the type of configuration we explored in a previous post.  

 

Checking Core Configuration

Before we move on to looking at the logs for the sas-certframe init container we will first check some of the Kubernetes items that should exist. To complete these steps, you will need to have a Kubernetes role that has access to read secrets, config maps, and certificates. Throughout these commands the variable ${NS} refers to the SAS Viya namespace.

 

  1. Ingress Certificate – use the following command to confirm the Ingress certificate pair has been loaded

     

    kubectl -n ${NS} describe secret `kubectl -n ${NS} get ing sas-logon-app \
-o=jsonpath={.spec.tls[*].secretName}`

     

    This command is actually two separate commands, one embedded inside the other. If we look at the inner command which is using kubectl to get the ingress definition for sas-logon-app. This then has an output (-o) option that means it returns just the secret name from the ingress definition. This secret name is passed to the outer command that uses kubectl to describe the named secret.

    You should see something like:

     

    Name:         sas-ingress-certificate-cf44kkkfg2
Namespace:    gelenv-stable
Labels:       sas.com/admin=cluster-local
              sas.com/deployment=sas-viya
Annotations:
Type:         kubernetes.io/tls

Data
====
tls.crt:  5129 bytes
tls.key:  1679 bytes

     

    This shows first that the Ingress definition correctly identifies the customer’s provided certificate pair, since the secret name is returned from the Ingress definition. Then it shows that the tls.key and tls.crt have been correctly loaded into Kubernetes, since they have a size greater than 0 bytes.

  2. Confirm that the Certificate Authority (CA), if required, has been loaded into a Kubernetes config map with the following: 
    kubectl -n ${NS} describe configmap sas-customer-provided-ca-certificates

     

    You should see something like the following: 
    Name:         sas-customer-provided-ca-certificates-g5cgm68tkg
Namespace:    gelenv-stable
Labels:       sas.com/admin=cluster-local
              sas.com/deployment=sas-viya
Annotations:
Data
====
ca.crt:
----
Bag Attributes: <Empty Attributes>
subject=/DC=com/DC=sas/CN=SAS SHA2 Root CA
issuer=/DC=com/DC=sas/CN=SAS SHA2 Root CA
-----BEGIN CERTIFICATE-----
....
....
-----END CERTIFICATE-----

Events:  <none>

     

  3. Check a Kubernetes certificate, in this case the certificate for sas-consul-server-0, to ensure the correct cert-manager issuer is defined against it: 
    echo "Issuer is: "`kubectl -n ${NS} get certificate sas-consul-server-0 \
-o=jsonpath='{.spec.issuerRef.name}'`

     

    You should see the following: 
    >Issuer is: sas-viya-issuer

     

    This shows the correct sas-viya-issuer assigned to the Kubernetes certificate.

  4. Check a Kubernetes certificate and check the last status message, again in this case for the sas-consul-server-0:

    echo "Last status message is: "`kubectl -n ${NS} get certificate sas-consul-server-0 \
-o=jsonpath='{.status.conditions[*].message}'`

     

    You should see something like the following: 
    >Last status message is: Certificate is up to date and has not expired

     

    This shows that the certificate is valid for use.

  5. Check a Kubernetes secret, in this case for sas-consul-server-0, to ensure the correct cert-manager issuer is defined against it:
    >echo "Issuer is: "`kubectl -n ${NS} get secrets sas-consul-server-0 \
-o=jsonpath='{.metadata.annotations.cert-manager\.io/issuer-name}'`

     

    You should see the following:
     Issuer is: sas-viya-issuer
    This validates that the Kubernetes secret has been generated by the expected sas-viya-issuer.

  6. List all the details of the Kubernetes secret associated with one of the SAS Consul pods: 
    >kubectl -n ${NS} describe secrets sas-consul-server-0

     

    You should see something like the following: 
    Name:         sas-consul-server-0
Namespace:    gelenv-stable
Labels:       <none>
Annotations:  cert-manager.io/alt-names:
                gelenv-stable.rext03-0153.race.sas.com,intnode02,localhost,sas-consul-server,sas-consul-server-0,sas-consul-server-0.sas-consul-server-int...
              cert-manager.io/certificate-name: sas-consul-server-0
              cert-manager.io/common-name: sas-consul-server-0
              cert-manager.io/ip-sans: 10.42.2.173,10.43.52.161,127.0.0.1,192.168.6.2
              cert-manager.io/issuer-group:
              cert-manager.io/issuer-kind: Issuer
              cert-manager.io/issuer-name: sas-viya-issuer
              cert-manager.io/uri-sans:

Type:  kubernetes.io/tls

Data
====
ca.crt:   1135 bytes
tls.crt:  1501 bytes
tls.key:  1679 bytes

     

    This again shows the correct issuer and shows the certificate pair is correctly loaded into the Kubernetes secret, since ca.crt, tls.crt and tls.key are greater than 0 bytes.

  7. Finally, we can combine kubectl and OpenSSL to show the contents of the signed server identity certificate for the sas-consul-server-0 from the Kubernetes secret:

    kubectl -n ${NS} get secret sas-consul-server-0 \
-o=jsonpath='{.data.tls\.crt}'| base64 -d |openssl x509 \
-noout -text -certopt ca_default,no_sigdump

     

    We use kubectl to get the secret for sas-consul-server-0 and specify the output to be just the tls.crt data. This is base64 decoded and then piped into OpenSSL.

    You should see something like the following: 
            Serial Number:
            cf:f3:18:de:6c:c5:1f:f5:55:99:ab:c3:eb:71:ba:76
    Signature Algorithm: sha256WithRSAEncryption
        Validity
            Not Before: Nov  7 07:36:02 2020 GMT
            Not After : Nov  7 07:36:02 2022 GMT
        Subject: O=SAS Institute, CN=sas-consul-server-0
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage:
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Authority Key Identifier:
                keyid:F8:B2:86:04:82:A0:E2:76:D3:76:73:A5:F5:2C:22:BD:13:6F:D0:63

            X509v3 Subject Alternative Name:
                DNS:gelenv-stable.pdcesx21089.race.sas.com, DNS:intnode04, DNS:localhost, DNS:sas-consul-server, DNS:sas-consul-server-0, DNS:sas-consul-server-0.sas-consul-server-internal.gelenv-stable.svc.cluster.local, IP Address:10.42.3.7, IP Address:10.43.182.68, IP Address:127.0.0.1, IP Address:192.168.6.4

     

    Due to the flags on the OpenSSL command we suppress the full certificate output and only show the subject, validity dates, and X509v3 extensions including the Subject Alternative Names (SAN). Which line up with annotations on the Kubernetes secret object we looked at in the previous output.

 

This completes a basic check of the different Kubernetes items that should be created. If any of these items do not exist you need to review the kustomization.yaml and the site.yaml generated from it.

 

Reviewing Init Container Logs

Now that we have checked that the correct items exist, we can review the logs for the sas-certframe init container. This init container will be run in all SAS pods within the namespace. Only if the init container completes successfully will the main service container start within the pod. This means the SAS Viya environment will not start correctly until the sas-certframe init container has completed. So, we will take one of the key services and examine the sas-certframe init container log from that pod. Use the following command to pipe the log to a local file:

 

kubectl -n ${NS} logs sas-consul-server-0 sas-certframe > /tmp/sas-certframe.log

 

This will result in the log messages for sas-certframe being written to the /tmp/sas-certframe.log file. This will make it easier to review the log. The resulting file can be copied around and then opened in your preferred text editor. We can now walk through the log.

 

The first section of the log will contain something like the following:

 

2020-11-06 10:42:51 - [INFO]  - KUBE_TOKEN is not blank
2020-11-06 10:42:51 - [INFO]  - KUBE_NAMESPACE = gelenv-stable
2020-11-06 10:42:51 - [INFO]  - KUBE_CACERT = /certframe-token/ca.crt
2020-11-06 10:42:51 - [INFO]  - KUBERNETES_SERVICE_HOST = 10.43.0.1
2020-11-06 10:42:51 - [INFO]  - KUBERNETES_PORT_443_TCP_PORT = 443
2020-11-06 10:42:51 - [INFO]  - KUBE_POD_NAME = sas-consul-server-0
2020-11-06 10:42:51 - [WARN]  - SAS_SECURITY_ARTIFACTS_DIR not set
2020-11-06 10:42:51 - [INFO]  - SAS_CA_CERTIFICATE_SECRET_NAME = sas-viya-ca-certificate-secret
2020-11-06 10:42:51 - [INFO]  - SAS_INGRESS_CERTIFICATE_SECRET_NAME = sas-ingress-certificate-cf44kkkfg2
2020-11-06 10:42:51 - [INFO]  - SAS_ADDITIONAL_CA_CERTIFICATES_DIR = /customer-provided-ca-certificates
2020-11-06 10:42:52 - [INFO]  - SAS_TRUSTED_CA_CERTIFICATES_PEM_FILE = /security/trustedcerts.pem
2020-11-06 10:42:52 - [WARN]  - SAS_TRUSTED_CA_CERTIFICATES_JKS_FILE not set
2020-11-06 10:42:52 - [INFO]  - SAS_CERTIFICATE_CA_CERTIFICATE_FILE = /security/ca.crt
2020-11-06 10:42:52 - [INFO]  - SAS_CERTIFICATE_PRIVATE_KEY_FILE = /security/tls.key
2020-11-06 10:42:52 - [WARN]  - SAS_CERTIFICATE_PRIVATE_KEY_PERMISSIONS not set
2020-11-06 10:42:52 - [WARN]  - SAS_CERTIFICATE_PRIVATE_KEY_PASSPHRASE_FILE not set
2020-11-06 10:42:52 - [INFO]  - SAS_CERTIFICATE_FILE = /security/tls.crt
2020-11-06 10:42:52 - [WARN]  - SAS_CERTIFICATE_SECRET_NAME not set; defaulting to sas-consul-server-0
2020-11-06 10:42:52 - [INFO]  - SAS_CERTIFICATE_GENERATOR = cert-manager
2020-11-06 10:42:52 - [WARN]  - SAS_CERTIFICATE_COMMON_NAME not set; defaulting to sas-consul-server-0.sas-consul-server-internal.gelenv-stable.svc.cluster.local
2020-11-06 10:42:52 - [INFO]  - SAS_CERTIFICATE_ISSUER = sas-viya-issuer
2020-11-06 10:42:52 - [WARN]  - SAS_CERTIFICATE_SAN_DNS not set
2020-11-06 10:42:52 - [WARN]  - SAS_CERTIFICATE_SAN_IP not set
2020-11-06 10:42:52 - [WARN]  - SAS_CERTIFICATE_ADDITIONAL_SAN_DNS not set
2020-11-06 10:42:52 - [WARN]  - SAS_CERTIFICATE_ADDITIONAL_SAN_IP not set
2020-11-06 10:42:52 - [INFO]  - SAS_CERTIFICATE_INGRESS_SAN_DNS = gelenv-stable.rext03-0025.race.sas.com
2020-11-06 10:42:52 - [WARN]  - SAS_CERTIFICATE_DURATION not set; defaulting to 17520h (2 years)
2020-11-06 10:42:52 - [WARN]  - SAS_CERTIFICATE_RENEWAL_LEAD_TIME not set; defaulting to 5m
2020-11-06 10:42:52 - [WARN]  - SAS_SSH_SECRET_NAME not set
2020-11-06 10:42:52 - [WARN]  - SAS_KEYS_SECRET_NAME not set
2020-11-06 10:42:52 - [WARN]  - SAS_KEYS_KEY_NAMES not set
2020-11-06 10:42:52 - [WARN]  - SAS_KEYS_KEY_TYPES not set; all keys will default to type uuid
2020-11-06 10:42:52 - [WARN]  - SAS_SECURITY_ARTIFACTS_DIR is unset, attempting to determine a safe temporary directory location

 

This shows the container picking up several required environment variables. Some specific environment variables to call out are:

 

  • SAS_CA_CERTIFICATE_SECRET_NAME
  • SAS_INGRESS_CERTIFICATE_SECRET_NAME
  • SAS_CERTIFICATE_ISSUER
  • SAS_CERTIFICATE_FILE
  • SAS_CERTIFICATE_PRIVATE_KEY_FILE
  • SAS_CERTIFICATE_CA_CERTIFICATE_FILE

 

The first three identify the Kubernetes secret names for the CA certificate and Ingress Certificate as well as the cert-manager issuer name. The last three we called out specify the filename and path of the certificate pair (private key and signed identity server certificate) and CA certificate.These three files will be created by sas-certframe.

 

One final environment variable to call out provides the filename and path of the trust store that sas-certframe will create. Depending on the SAS service this will be either SAS_TRUSTED_CA_CERTIFICATES_PEM_FILE or SAS_TRUSTED_CA_CERTIFICATES_JKS_FILE. You can see above in our case for sas-consul-server-0 we will require a PEM format trust store. While if we examine the sas-certframe log from a pod containing a Java SAS service it would be the second environment variable that is populated with the filename and path of the Java Key Store (JKS).

 

The first task that sas-certframe will perform is to build the trust store. To do this sas-certframe needs to obtain additional certificates to add to the Mozilla bundle. The next part of the log shows sas-certframe obtaining the Ingress Certificate from the Kubernetes secret provided by the environment variable SAS_INGRESS_CERTIFICATE_SECRET_NAME. An example is shown here:

 

2020-11-06 10:42:52 - [INFO]  - Configuring the trust stores
2020-11-06 10:42:52 - [INFO]  - Attempting to get the Ingress certificate from Kubernetes secret
2020-11-06 10:42:52 - [INFO]  - Gathering details about the ingress certificate from Kubernetes secret: sas-ingress-certificate-cf44kkkfg2
2020-11-06 10:42:52 - [INFO]  - Retrieving ingress certificate from tls.crt field
2020-11-06 10:42:52 - [INFO]  - Retrieved ingress certificate containing:
Bag Attributes
    localKeyID: 01 00 00 00 
    friendlyName: gelenv-stable.rext03-0025.race.sas.com
subject=/C=US/ST=North Carolina/L=Cary/O=SAS Institute Inc./OU=GEL/CN=gelenv-stable.rext03-0025.race.sas.com/emailAddress=stuart.rogers@sas.com
issuer=/DC=com/DC=SAS/CN=SAS SHA2 Issuing CA02
-----BEGIN CERTIFICATE-----
...
...
-----END CERTIFICATE-----
Bag Attributes: <Empty Attributes>
subject=/DC=com/DC=SAS/CN=SAS SHA2 Issuing CA02
issuer=/DC=com/DC=sas/CN=SAS SHA2 Root CA
-----BEGIN CERTIFICATE-----
...
...
-----END CERTIFICATE-----
2020-11-06 10:42:52 - [INFO]  - Retrieving the ingress CA certificate chain from ca.crt field
Traceback (most recent call last):
  File "<string>", line 1, in <module>
KeyError: 'ca.crt'
2020-11-06 10:42:52 - [WARN]  - Unable to retrieve the ingress ca.crt from secret: sas-ingress-certificate-cf44kkkfg2
2020-11-06 10:42:52 - [WARN]  - The ingress certificate may not have a certificate chain configured: this may cause issues with certificate trust

 

Once sas-certframe has obtained the certificate chain for the Ingress Controller it will use the environment variable SAS_CA_CERTIFICATE_SECRET_NAME to identify the CA certificate valid for the SAS Viya environment. The sas-certframe container will then load the CA certificate from the Kubernetes secret, as shown here:

 

2020-11-06 10:42:52 - [INFO]  - Attempting to get the Issuing CA certificate from Kubernetes secret
2020-11-06 10:42:52 - [INFO]  - Gathering details about the issuer CA certificate from Kubernetes secret: sas-viya-ca-certificate-secret
2020-11-06 10:42:52 - [INFO]  - Retrieving issuer CA certificate from tls.crt field
2020-11-06 10:42:52 - [INFO]  - Retrieved issuer CA certificate containing:
-----BEGIN CERTIFICATE-----
...
...
-----END CERTIFICATE-----
2020-11-06 10:42:52 - [INFO]  - Retrieving the issuer CA certificate chain from ca.crt field
2020-11-06 10:42:52 - [INFO]  - Retrieved issuer CA certificate chain containing:
-----BEGIN CERTIFICATE-----
...
...
-----END CERTIFICATE-----
2020-11-06 10:42:53 - [INFO]  - Issuer CA certificate is self-signed
2020-11-06 10:42:53 - [INFO]  - The issuer CA certificate is already a full chain; ignoring issuer CA certificate chain

 

Now that sas-certframe has obtained both the Ingress Controller certificate chain and CA certificate these can be added to the Mozilla bundle to start the build of the trust store for the service. The log messages shown below provided details of each certificate being added to the trust store:

 

2020-11-06 10:42:53 - [INFO]  - SAS_TRUSTED_CA_CERTIFICATES_PEM_FILE environment variable is set to /security/trustedcerts.pem
2020-11-06 10:42:53 - [INFO]  - Creating PEM trust bundle
2020-11-06 10:42:53 - [INFO]  - Attempting to get working copy of PEM trustbundle from /opt/sas/viya/home/SASSecurityCertificateFramework/tls/certs/ca-bundle.pem
2020-11-06 10:42:53 - [INFO]  - Adding full issuer CA certificate chain to Mozilla PEM trust bundle using sas-crypto-management
level=info app=sas-crypto-management timestamp=2020-11-06T10:42:53.319459424Z function=trust_store.logCertificateOperation msg="Adding certificate to PEM trustStore" certificateIterator="1 of 1" trustStoreFileName=/security/temp/trustbundles/ca-bundle.pem certificateIdentifier="CN=sas-viya-root-ca-certificate/OU=/O=/L=/S=/C="
level=info app=sas-crypto-management timestamp=2020-11-06T10:42:53.342518202Z function=trust_store.truststoreAddPemCertToPem msg="Successfully read PEM trustStore input file" trustStoreFileName=/security/temp/trustbundles/ca-bundle.pem cb.Length=138
level=info app=sas-crypto-management timestamp=2020-11-06T10:42:53.342572212Z function=trust_store.truststoreAddPemCertToPem msg="Obtained identifier for to-be-added certificate" certificateIdentifier="CN=sas-viya-root-ca-certificate/OU=/O=/L=/S=/C=" trustStoreFileName=/security/temp/trustbundles/ca-bundle.pem
level=info app=sas-crypto-management timestamp=2020-11-06T10:42:53.342622348Z function=trust_store.truststoreAddPemCertToPem isCertificatePresentInTrustStore=false certificateIdentifier="CN=sas-viya-root-ca-certificate/OU=/O=/L=/S=/C=" trustStoreFileName=/security/temp/trustbundles/ca-bundle.pem
level=info app=sas-crypto-management timestamp=2020-11-06T10:42:53.342639427Z function=trust_store.truststoreAddPemCertToPem msg="Certificate NOT present, ADDING it" certificateIdentifier="CN=sas-viya-root-ca-certificate/OU=/O=/L=/S=/C=" trustStoreFileName=/security/temp/trustbundles/ca-bundle.pem
level=info app=sas-crypto-management timestamp=2020-11-06T10:42:53.342650349Z function=trust_store.truststoreAddPemCertToPem msg="Writing PEM trustStore output file" trustStoreFileName=/security/temp/trustbundles/ca-bundle.pem cb.Length=139
2020-11-06 10:42:53 - [INFO]  - Adding full ingress certificate chain to Mozilla PEM trust bundle using sas-crypto-management
level=info app=sas-crypto-management timestamp=2020-11-06T10:42:53.598937488Z function=trust_store.logCertificateOperation msg="Adding certificate to PEM trustStore" certificateIterator="1 of 2" trustStoreFileName=/security/temp/trustbundles/ca-bundle.pem certificateIdentifier="CN=gelenv-stable.rext03-0025.race.sas.com/OU=GEL/O=SAS Institute Inc./L=Cary/S=North Carolina/C=US"
level=info app=sas-crypto-management timestamp=2020-11-06T10:42:53.62780719Z function=trust_store.truststoreAddPemCertToPem msg="Successfully read PEM trustStore input file" trustStoreFileName=/security/temp/trustbundles/ca-bundle.pem cb.Length=139
level=info app=sas-crypto-management timestamp=2020-11-06T10:42:53.627883735Z function=trust_store.truststoreAddPemCertToPem msg="Obtained identifier for to-be-added certificate" certificateIdentifier="CN=gelenv-stable.rext03-0025.race.sas.com/OU=GEL/O=SAS Institute Inc./L=Cary/S=North Carolina/C=US" trustStoreFileName=/security/temp/trustbundles/ca-bundle.pem
level=info app=sas-crypto-management timestamp=2020-11-06T10:42:53.628071275Z function=trust_store.truststoreAddPemCertToPem isCertificatePresentInTrustStore=false certificateIdentifier="CN=gelenv-stable.rext03-0025.race.sas.com/OU=GEL/O=SAS Institute Inc./L=Cary/S=North Carolina/C=US" trustStoreFileName=/security/temp/trustbundles/ca-bundle.pem
level=info app=sas-crypto-management timestamp=2020-11-06T10:42:53.628103391Z function=trust_store.truststoreAddPemCertToPem msg="Certificate NOT present, ADDING it" certificateIdentifier="CN=gelenv-stable.rext03-0025.race.sas.com/OU=GEL/O=SAS Institute Inc./L=Cary/S=North Carolina/C=US" trustStoreFileName=/security/temp/trustbundles/ca-bundle.pem
level=info app=sas-crypto-management timestamp=2020-11-06T10:42:53.628123179Z function=trust_store.truststoreAddPemCertToPem msg="Writing PEM trustStore output file" trustStoreFileName=/security/temp/trustbundles/ca-bundle.pem cb.Length=140
level=info app=sas-crypto-management timestamp=2020-11-06T10:42:53.664235677Z function=trust_store.logCertificateOperation msg="Adding certificate to PEM trustStore" certificateIterator="2 of 2" trustStoreFileName=/security/temp/trustbundles/ca-bundle.pem certificateIdentifier="CN=SAS SHA2 Issuing CA02/OU=/O=/L=/S=/C="
level=info app=sas-crypto-management timestamp=2020-11-06T10:42:53.676896774Z function=trust_store.truststoreAddPemCertToPem msg="Successfully read PEM trustStore input file" trustStoreFileName=/security/temp/trustbundles/ca-bundle.pem cb.Length=140
level=info app=sas-crypto-management timestamp=2020-11-06T10:42:53.676929485Z function=trust_store.truststoreAddPemCertToPem msg="Obtained identifier for to-be-added certificate" certificateIdentifier="CN=SAS SHA2 Issuing CA02/OU=/O=/L=/S=/C=" trustStoreFileName=/security/temp/trustbundles/ca-bundle.pem
level=info app=sas-crypto-management timestamp=2020-11-06T10:42:53.677064055Z function=trust_store.truststoreAddPemCertToPem isCertificatePresentInTrustStore=false certificateIdentifier="CN=SAS SHA2 Issuing CA02/OU=/O=/L=/S=/C=" trustStoreFileName=/security/temp/trustbundles/ca-bundle.pem
level=info app=sas-crypto-management timestamp=2020-11-06T10:42:53.677084633Z function=trust_store.truststoreAddPemCertToPem msg="Certificate NOT present, ADDING it" certificateIdentifier="CN=SAS SHA2 Issuing CA02/OU=/O=/L=/S=/C=" trustStoreFileName=/security/temp/trustbundles/ca-bundle.pem
level=info app=sas-crypto-management timestamp=2020-11-06T10:42:53.677100357Z function=trust_store.truststoreAddPemCertToPem msg="Writing PEM trustStore output file" trustStoreFileName=/security/temp/trustbundles/ca-bundle.pem cb.Length=141

 

Now since the Root CA certificate and possibly any other certificates the customer wants to add to the trust stores are placed in a Kubernetes configmap these are also loaded into the trust store. The Kubernetes configmap is accessed as a volume mount. The following log messages show adding these additional CA certificates:

 

2020-11-06 10:42:53 - [INFO]  - Adding additional CA certificates from directory /customer-provided-ca-certificates
2020-11-06 10:42:53 - [INFO]  - Adding the CA certificate from: /customer-provided-ca-certificates/..2020_11_06_10_42_47.843262752/ca.crt
2020-11-06 10:42:53 - [INFO]  - Certificate content:
Bag Attributes: 
subject=/DC=com/DC=sas/CN=SAS SHA2 Root CA
issuer=/DC=com/DC=sas/CN=SAS SHA2 Root CA
-----BEGIN CERTIFICATE-----
...
...
-----END CERTIFICATE-----
level=info app=sas-crypto-management timestamp=2020-11-06T10:42:53.906909129Z function=trust_store.logCertificateOperation msg="Adding certificate to PEM trustStore" certificateIterator="1 of 1" trustStoreFileName=/security/temp/trustbundles/ca-bundle.pem certificateIdentifier="CN=SAS SHA2 Root CA/OU=/O=/L=/S=/C="
level=info app=sas-crypto-management timestamp=2020-11-06T10:42:53.925358468Z function=trust_store.truststoreAddPemCertToPem msg="Successfully read PEM trustStore input file" trustStoreFileName=/security/temp/trustbundles/ca-bundle.pem cb.Length=141
level=info app=sas-crypto-management timestamp=2020-11-06T10:42:53.925419165Z function=trust_store.truststoreAddPemCertToPem msg="Obtained identifier for to-be-added certificate" certificateIdentifier="CN=SAS SHA2 Root CA/OU=/O=/L=/S=/C=" trustStoreFileName=/security/temp/trustbundles/ca-bundle.pem
level=info app=sas-crypto-management timestamp=2020-11-06T10:42:53.925485201Z function=trust_store.truststoreAddPemCertToPem isCertificatePresentInTrustStore=false certificateIdentifier="CN=SAS SHA2 Root CA/OU=/O=/L=/S=/C=" trustStoreFileName=/security/temp/trustbundles/ca-bundle.pem
level=info app=sas-crypto-management timestamp=2020-11-06T10:42:53.925501428Z function=trust_store.truststoreAddPemCertToPem msg="Certificate NOT present, ADDING it" certificateIdentifier="CN=SAS SHA2 Root CA/OU=/O=/L=/S=/C=" trustStoreFileName=/security/temp/trustbundles/ca-bundle.pem
level=info app=sas-crypto-management timestamp=2020-11-06T10:42:53.925511779Z function=trust_store.truststoreAddPemCertToPem msg="Writing PEM trustStore output file" trustStoreFileName=/security/temp/trustbundles/ca-bundle.pem cb.Length=142
2020-11-06 10:42:53 - [INFO]  - Copying PEM trust store to requested SAS_TRUSTED_CA_CERTIFICATES_PEM_FILE: /security/trustedcerts.pem
2020-11-06 10:42:53 - [INFO]  - Finished adding certificates to the trust store

 

With the trust store constructed sas-certframe will move onto requesting the certificate pair (private key and signed server identity certificate) for the service. The next segment of the log shows sas-certframe building the request for the certificate pair:

 

2020-11-06 10:42:53 - [INFO]  - Requested certificate file: /security/tls.crt
2020-11-06 10:42:54 - [WARN]  - Variable SAS_CERTIFICATE_SECRET_NAME not set; defaulting to KUBE_POD_NAME value sas-consul-server-0
2020-11-06 10:42:54 - [INFO]  - Gathering details about the current pod
2020-11-06 10:42:54 - [WARN]  - Pod metadata has not yet been fully populated. Sleeping 1s to allow information to be populated.
2020-11-06 10:42:55 - [WARN]  - Pod metadata has not yet been fully populated. Sleeping 1s to allow information to be populated.
2020-11-06 10:42:56 - [WARN]  - Pod metadata has not yet been fully populated. Sleeping 1s to allow information to be populated.
2020-11-06 10:42:57 - [WARN]  - Pod metadata has not yet been fully populated. Sleeping 1s to allow information to be populated.
2020-11-06 10:42:58 - [WARN]  - Pod metadata has not yet been fully populated. Sleeping 1s to allow information to be populated.
2020-11-06 10:42:59 - [WARN]  - Pod metadata has not yet been fully populated. Sleeping 1s to allow information to be populated.
2020-11-06 10:43:01 - [WARN]  - Pod metadata has not yet been fully populated. Sleeping 1s to allow information to be populated.
2020-11-06 10:43:02 - [WARN]  - Pod metadata has not yet been fully populated. Sleeping 1s to allow information to be populated.
2020-11-06 10:43:03 - [WARN]  - Pod metadata has not yet been fully populated. Sleeping 1s to allow information to be populated.
2020-11-06 10:43:04 - [WARN]  - Pod metadata has not yet been fully populated. Sleeping 1s to allow information to be populated.
2020-11-06 10:43:05 - [WARN]  - Pod metadata has not yet been fully populated. Sleeping 1s to allow information to be populated.
2020-11-06 10:43:06 - [WARN]  - Pod metadata has not yet been fully populated. Sleeping 1s to allow information to be populated.
2020-11-06 10:43:07 - [WARN]  - Pod metadata has not yet been fully populated. Sleeping 1s to allow information to be populated.
2020-11-06 10:43:08 - [WARN]  - Pod metadata has not yet been fully populated. Sleeping 1s to allow information to be populated.
2020-11-06 10:43:10 - [WARN]  - Pod metadata has not yet been fully populated. Sleeping 1s to allow information to be populated.
2020-11-06 10:43:11 - [WARN]  - Pod metadata has not yet been fully populated. Sleeping 1s to allow information to be populated.
2020-11-06 10:43:12 - [WARN]  - Pod metadata has not yet been fully populated. Sleeping 1s to allow information to be populated.
2020-11-06 10:43:13 - [WARN]  - Pod metadata has not yet been fully populated. Sleeping 1s to allow information to be populated.
2020-11-06 10:43:14 - [INFO]  - Requested certificate generator: cert-manager
2020-11-06 10:43:14 - [INFO]  - Gathering information to determine contents of SAN fields
2020-11-06 10:43:14 - [INFO]  - Determined this pod's IP address to be: 10.42.0.70
2020-11-06 10:43:14 - [INFO]  - Determined this pod's node name to be: intnode03
2020-11-06 10:43:14 - [INFO]  - Determined this pod's host IP address to be: 192.168.6.3
2020-11-06 10:43:14 - [INFO]  - Determined app.kubernetes.io/name to be: sas-consul-server
2020-11-06 10:43:14 - [INFO]  - Gathering details about services associated with this pod based on the app.kuberenetes.io/name label
2020-11-06 10:43:14 - [INFO]  - Determined that this pod has the following associated service names: sas-consul-server
2020-11-06 10:43:14 - [INFO]  - Determined that the associated service names have the following ClusterIPs: 10.43.95.162
2020-11-06 10:43:14 - [INFO]  - SAS_CERTIFICATE_SAN_DNS is currently: 
2020-11-06 10:43:14 - [INFO]  - Concatenating associated service names into SAS_CERTIFICATE_SAN_DNS, if any
2020-11-06 10:43:15 - [INFO]  - Adding hostname -f and hostname -s to SAS_CERTIFICATE_SAN_DNS
2020-11-06 10:43:15 - [INFO]  - Adding KUBE_POD_NAME sas-consul-server-0 to SAS_CERTIFICATE_SAN_DNS
2020-11-06 10:43:15 - [INFO]  - Adding KUBE_NODE_NAME intnode03 to SAS_CERTIFICATE_SAN_DNS
2020-11-06 10:43:15 - [INFO]  - Adding localhost to SAS_CERTIFICATE_SAN_DNS
2020-11-06 10:43:15 - [INFO]  - Adding SAS_CERTIFICATE_INGRESS_SAN_DNS to SAS_CERTIFICATE_SAN_DNS
2020-11-06 10:43:15 - [INFO]  - New SAS_CERTIFICATE_SAN_DNS: gelenv-stable.rext03-0025.race.sas.com intnode03 localhost sas-consul-server sas-consul-server-0 sas-consul-server-0.sas-consul-server-internal.gelenv-stable.svc.cluster.local
2020-11-06 10:43:15 - [INFO]  - SAS_CERTIFICATE_SAN_IP is currently: 
2020-11-06 10:43:15 - [INFO]  - Concatenating service IP addresses into SAS_CERTIFICATE_SAN_IP, if any
2020-11-06 10:43:15 - [INFO]  - Concatenating KUBE_POD_IP 10.42.0.70 into SAS_CERTIFICATE_SAN_IP
2020-11-06 10:43:15 - [INFO]  - Concatenating KUBE_HOST_IP 192.168.6.3 into SAS_CERTIFICATE_SAN_IP
2020-11-06 10:43:15 - [INFO]  - Concatenating 127.0.0.1 into SAS_CERTIFICATE_SAN_IP
2020-11-06 10:43:15 - [INFO]  - New SAS_CERTIFICATE_SAN_IP: 10.42.0.70 10.43.95.162 127.0.0.1 192.168.6.3
2020-11-06 10:43:15 - [INFO]  - Current pod uid is: 37c4a785-d6f3-4719-a0f4-3139bd4f9ccc
2020-11-06 10:43:15 - [WARN]  - SAS_CERTIFICATE_COMMON_NAME not set
2020-11-06 10:43:15 - [WARN]  - hostname -f is longer than 64 characters and will be rejected; instead, defaulting to 'hostname -s' value sas-consul-server-0
2020-11-06 10:43:15 - [WARN]  - SAS_CERTIFICATE_DURATION not set; defaulting to 17520h (2 years)
2020-11-06 10:43:15 - [WARN]  - SAS_CERTIFICATE_RENEWAL_LEAD_TIME not set; defaulting to 5m
2020-11-06 10:43:15 - [INFO]  - Using cert-manager to issue certificate for pod
2020-11-06 10:43:15 - [INFO]  - Creating JSON request for certificate to be stored in secret: sas-consul-server-0
2020-11-06 10:43:15 - [INFO]  - Relevant metadata:
2020-11-06 10:43:15 - [INFO]  -       SAS_CERTIFICATE_SECRET_NAME: sas-consul-server-0
2020-11-06 10:43:15 - [INFO]  -            SAS_CERTIFICATE_ISSUER: sas-viya-issuer
2020-11-06 10:43:15 - [INFO]  -       SAS_CERTIFICATE_COMMON_NAME: sas-consul-server-0
2020-11-06 10:43:15 - [INFO]  -           SAS_CERTIFICATE_SAN_DNS: gelenv-stable.rext03-0025.race.sas.com intnode03 localhost sas-consul-server sas-consul-server-0 sas-consul-server-0.sas-consul-server-internal.gelenv-stable.svc.cluster.local
2020-11-06 10:43:15 - [INFO]  -            SAS_CERTIFICATE_SAN_IP: 10.42.0.70 10.43.95.162 127.0.0.1 192.168.6.3
2020-11-06 10:43:15 - [INFO]  -          SAS_CERTIFICATE_DURATION: 17520h
2020-11-06 10:43:15 - [INFO]  - SAS_CERTIFICATE_RENEWAL_LEAD_TIME: 5m
2020-11-06 10:43:15 - [INFO]  - Created JSON request containing:
{"kind": "Certificate", "spec": {"usages": ["digital signature", "key encipherment", "server auth", "client auth"], "issuerRef": {"kind": "Issuer", "name": "sas-viya-issuer"}, "dnsNames": ["gelenv-stable.rext03-0025.race.sas.com", "intnode03", "localhost", "sas-consul-server", "sas-consul-server-0", "sas-consul-server-0.sas-consul-server-internal.gelenv-stable.svc.cluster.local"], "commonName": "sas-consul-server-0", "renewBefore": "5m", "ipAddresses": ["10.42.0.70", "10.43.95.162", "127.0.0.1", "192.168.6.3"], "secretName": "sas-consul-server-0", "duration": "17520h", "organization": ["SAS Institute"]}, "apiVersion": "cert-manager.io/v1alpha2", "metadata": {"ownerReferences": [{"kind": "Pod", "uid": "37c4a785-d6f3-4719-a0f4-3139bd4f9ccc", "apiVersion": "v1", "controller": true, "blockOwnerDeletion": false, "name": "sas-consul-server-0"}], "name": "sas-consul-server-0"}}
2020-11-06 10:43:15 - [INFO]  - Checking to see if certificate object already exists

 

The key part we’ll want to check is the section for "Relevant metadata"; this shows

  • the Kubernetes secret name the certificate pair will be stored in
  • the cert-manager issuer that will be used
  • the common name of the certificate
  • the Subject Alternate Name (SAN) DNS entries
  • the Subject Alternate Name (SAN) IP entries
  • the certificate lifetime

 

Next in the log, we can see the certificate request submitted and sas-certframe waiting for the certificate request to be signed:

 

2020-11-06 10:43:15 - [INFO]  - Creating certificate object
2020-11-06 10:43:16 - [INFO]  - The certificate object was successfully created
2020-11-06 10:43:16 - [INFO]  - New object resource version: 3219583
2020-11-06 10:43:16 - [INFO]  - Waiting for certificate to show a new status of 'Ready'
2020-11-06 10:43:16 - [INFO]  - Will wait up to 300 seconds
2020-11-06 10:43:16 - [WARN]  - Certificate is not yet showing a status of 'Ready'
2020-11-06 10:43:16 - [WARN]  - Certificate is not yet ready for consumption; sleeping 2 seconds before trying again
...
...
...
...
2020-11-06 10:46:17 - [INFO]  - Certificate is showing a status of 'Ready'
2020-11-06 10:46:17 - [INFO]  - Certificate object has a new last transition time: 2020-11-06T10:46:14Z
2020-11-06 10:46:17 - [INFO]  - Certificate is ready for consumption

 

Depending on how busy cert-manager is, the request can take several minutes to complete. In our abridged log above, we can see it took 3 minutes for the certificate pair to be ready. If it takes more than 300 seconds, the sas-certframe container will exit with an error, non-zero return code. So, Kubernetes will restart the sas-certframe container and it will continue where it left off, waiting for the certificate to have a status of "Ready".

 

Finally, sas-certframe will retrieve the certificate pair from the Kubernetes secret and make it available to the SAS service container, as shown below:

 

2020-11-06 10:46:17 - [INFO]  - Certificate retrieved successfully
2020-11-06 10:46:18 - [INFO]  - Certificate Information:
Issuer: CN=sas-viya-root-ca-certificate/OU=/O=/L=/S=/C=
Subject: CN=sas-consul-server-0/OU=/O=SAS Institute/L=/S=/C=
Valid from: Fri Nov  6 10:45:23 UTC 2020
Valid until: Sun Nov  6 10:45:23 UTC 2022
CA: false
SAN DNS: gelenv-stable.rext03-0025.race.sas.com
SAN DNS: intnode03
SAN DNS: localhost
SAN DNS: sas-consul-server
SAN DNS: sas-consul-server-0
SAN DNS: sas-consul-server-0.sas-consul-server-internal.gelenv-stable.svc.cluster.local
No SAN Email entries
SAN IP: 10.42.0.70
SAN IP: 10.43.95.162
SAN IP: 127.0.0.1
SAN IP: 192.168.6.3
Serial Number: 6FFB5F409548A882653CCFA691EA35E5
SHA1 Fingerprint: E9D5B72ADF3F7539A392B8BB35DAC005E44739D7

*******************************************
*******************************************
2020-11-06 10:46:18 - [INFO]  - Validating that all requested SAN DNS fields are present
2020-11-06 10:46:18 - [INFO]  - Successfully validated all SAN DNS fields
2020-11-06 10:46:18 - [INFO]  - Validating that all requested SAN IP fields are present
2020-11-06 10:46:18 - [INFO]  - Successfully validated all SAN IP fields
2020-11-06 10:46:18 - [INFO]  - Retrieving private key and CA chain from newly generated server certificate
2020-11-06 10:46:18 - [INFO]  - Retrieved server certificate:
-----BEGIN CERTIFICATE-----
...
...
-----END CERTIFICATE-----
2020-11-06 10:46:18 - [INFO]  - Retrieved server CA certificate chain:
-----BEGIN CERTIFICATE-----
...
...
-----END CERTIFICATE-----
2020-11-06 10:46:18 - [INFO]  - Determining desired certificate format
2020-11-06 10:46:18 - [INFO]  - Retrieving the annotation sas.com/certificate-file-format from the pod
2020-11-06 10:46:18 - [INFO]  - SAS_CERTIFICATE_FILE_FORMAT is: pem
2020-11-06 10:46:18 - [WARN]  - No SAS_CERTIFICATE_PRIVATE_KEY_PASSPHRASE_FILE environment variable set; will not encrypt the private key
2020-11-06 10:46:18 - [INFO]  - Copying server cert to requested path: /security/tls.crt
2020-11-06 10:46:18 - [INFO]  - Copying server key to requested path: /security/tls.key
2020-11-06 10:46:18 - [INFO]  - Copying key to /security/tls.key
2020-11-06 10:46:18 - [INFO]  - Copying server certificate CA chain file to requested path /security/ca.crt
2020-11-06 10:46:18 - [INFO]  - Deleting temp directory: /security/temp
2020-11-06 10:46:18 - [INFO]  - Init container completed successfully
2020-11-06 10:46:18 - [INFO]  - Exiting with return code 0

 

The sas-certframe container successfully obtains the certificate pair and validates the Subject Alternate Name DNS and IP fields. The signed server identity certificate is written to /security/tls.crt, the private key is written to /security/tls.key and the CA certificate chain is written to /security/ca.crt. The /security mount point will be available to SAS service container running inside the same pod as the sas-certframe init container. If the SAS services is a Java service, the PEM format content will be added to a Java Key Store (JKS) file to make it available to the Java service. Once the certificate pair is written the sas-certframe init container completes with a return code of zero. Only then will the actual SAS service container launch within the pod.  

 

Closing Remarks

In this blog we’ve looked at some of the kubectl commands you can use to inspect the TLS related objects and options that will be set within the SAS Viya Kubernetes namespace. These commands will allow you to quickly check that the correct objects have been created in your deployment.

 

Then we moved on and reviewed the log output from the sas-certframe init container. Remember, there will be a sas-certframe init container run in every SAS Viya pod within your deployment. None of the SAS services will be able to start until the init container has completed. By reviewing the log for the sas-certframe init container we have shown you the correct steps the container must complete for a full-stack TLS deployment.

 

Note: This article addresses features of SAS Viya 2020.1 (and later).

Version history
Last update:
‎12-15-2020 02:13 PM
Updated by:
SAS Employee
Contributors

hackathon24-white-horiz.png

The 2025 SAS Hackathon Kicks Off on June 11!

Watch the live Hackathon Kickoff to get all the essential information about the SAS Hackathon—including how to join, how to participate, and expert tips for success.

YouTube LinkedIn

SAS AI and Machine Learning Courses

The rapid growth of AI technologies is driving an AI skills gap and demand for AI talent. Ready to grow your AI literacy? SAS offers free ways to get started for beginners, business leaders, and analytics professionals of all skill levels. Your future self will thank you.

Get started

Article Labels
Article Tags