Troubleshooting HTTPS Certificate Issues in SAS Viya: Untrusted certificates

In our previous post, we examined the issue of expired certificates and the steps required to identify and resolve them in your SAS Viya platform. Now, we turn our attention to another common SSL-related problem: untrusted certificates. When a certificate is deemed untrusted, it could disrupt essential communication paths or even compromise the security of the environment. In this post, we'll explore how to identify symptoms of untrusted certificates, understand the technical diagnosis, and resolve the issue by properly importing CA certificates into the appropriate trust stores.

Note: The $viya environment variable is used on kubectl commands that require the --namespace flag. Make sure that the environment variable is created before you use these kubectl commands.

Symptoms connecting via a browser

The most common symptom of an untrusted certificate is a browser warning when accessing the SAS Viya interface. This warning typically states that the connection is not secure, and you may encounter errors such as:

"Your connection is not private" in Chrome
"Warning: Potential Security Risk Ahead" in Firefox

These browser messages often point to a certificate that the browser doesn't trust because it's either self-signed, expired, or issued by an unknown authority.

Here is an example of the Error displayed if you were using Chrome:

Select any image to see a larger version.
Mobile users: To view the images, select the "Full" version at the bottom of the page.

In Kubernetes, internal services may also encounter untrusted certificate errors when trying to communicate with external systems. These errors can appear in logs with messages such as x509: certificate signed by unknown authority or SSL: CERTIFICATE_VERIFY_FAILED. This is especially common when dealing with external integrations like LDAP, Docker registries, or other third-party APIs.

Symptoms connecting via SWAT

When using the SWAT package to access CAS on a SAS Viya platform, you may encounter SSL-related issues that prevent secure communication with the CAS server. One common error is the requests.exceptions.SSLError, which indicates that the certificate verification process has failed during an HTTPS connection attempt. This error typically occurs when the client cannot verify the server's SSL certificate, often due to missing or untrusted certificates in the local trust store. Here are the three specific instances of this error depending on the way you try to access your platform:

If you are trying to access the REST interface via the Ingress:

requests.exceptions.SSLError: HTTPSConnectionPool(host='fradae-p02059-rg.gelenable.sas.com', port=443): Max retries exceeded with url: /cas-shared-default-http/cas/sessions (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)'),))

If you are trying to access the REST interface directly (if you enable the external access):

requests.exceptions.SSLError: HTTPSConnectionPool(host='4.236.0.232', port=8777): Max retries exceeded with url: /cas/sessions (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)'),))

If you are trying to access the binary interface directly (if you enable the external access):

ERROR: The TCP/IP negClientSSL support routine failed with status 807ff008, hostname 4.236.0.143 port 5570
ERROR: Certificate is incorrect for the specified server.
ERROR: Encryption run-time execution error
ERROR: Failed to connect to host '4.236.0.143', port 5570.

Technical Diagnosis

The root cause of these untrusted certificate issues usually comes down to either the certificate not being issued by a trusted certificate authority (CA) or the trust chain being incomplete. When an internal pod tries to connect to an external service, it relies on a trusted CA to verify the legitimacy of the certificate presented by that service. If the CA is missing or not trusted, the connection will fail.

The diagnostic process typically involves checking the certificate chain for the service in question. Using tools like `openssl` or Kubernetes logs, you can inspect the certificate to verify if it's signed by a trusted CA or if it's missing intermediate certificates in the chain. In the case of browser warnings, examining the certificate details within the browser's security tab can give you clues about why it is flagged as untrusted.

Run these commands from a linux server that does have access to your Kubernetes cluster. It must have the kubectl and the openssl command line tools installed.

INGRESS_HOST=$(kubectl -n $viya get $(kubectl -n $viya get configmap --output name | grep ingress-input) -o jsonpath='{.data.INGRESS_HOST}'); \
echo X | openssl s_client -connect ${INGRESS_HOST}:443 -servername $INGRESS_HOST 2>/dev/null | openssl verify

In this case, you should see the following output:

CN = sas-viya-openssl-ingress-certificate
error 20 at 0 depth lookup:unable to get local issuer certificate

Resolution

When connecting to the Ingress

Once you've identified that the certificate is untrusted, the next step is to obtain a copy of the Root CA certificate. You can download this directly from the browser (if it has been included) or request it from the service administrator.

If the Root CA has been included into the certificate chain, here is how to extract it, run these commands from a linux server that does have access to your Kubernetes cluster. It must have the kubectl and the openssl command line tools installed.

INGRESS_HOST=$(kubectl -n $viya get $(kubectl -n $viya get configmap --output name | grep ingress-input) -o jsonpath='{.data.INGRESS_HOST}'); \
openssl s_client -showcerts -verify 5 -connect ${INGRESS_HOST}:8777 -servername $INGRESS_HOST < /dev/null | awk '/BEGIN CERTIFICATE/,/END CERTIFICATE/{ if(/BEGIN CERTIFICATE/){a++}; out="cert"a".pem"; print >out}'
# Rename each certificate with his subject name
for cert in *.pem; do
  newname=$( openssl x509 -in "$cert" -noout -subject | awk -F'CN = ' '{print $2}' | awk -F',' '{print $1}' | tr -d ' ' | tr '[:upper:]' '[:lower:]').pem
  echo "${newname}"; mv "${cert}" "${newname}"
done

You can graphically identify the Root CA issuer using the Certificate Viewer on Chrome.

Select any image to see a larger version.
Mobile users: To view the images, select the "Full" version at the bottom of the page.

Now, if you need to install the certificate in the Windows system’s trusted root CA store.

Here's how to do it:

Open the Certificate Management Console:
- Press Win + R, type mmc, and press Enter.
- In the MMC console, go to "File" > "Add/Remove Snap-in...".
- Select "Certificates" and click "Add".
- Choose "Computer account" and "Local computer", then click "Finish".
- Import the Certificate:
In the "Certificates" window, navigate to "Trusted Root Certification Authorities" > "Certificates".
- Right-click and select "All Tasks" > "Import".
- Browse for the .pem file and import it.
Confirm the Installation:
- Ensure the certificate appears in the list of trusted root CAs.

Now, if you need to add the certificate to the Linux system-wide trust store (RHEL as an example)

Copy the Certificate to the Appropriate Directory:
- Copy the certificate file (in .pem format) to /etc/pki/ca-trust/source/anchors/:
Update the CA Trust Store:
- Run the following command to update the system’s trust store:
- ```
sudo update-ca-trust
```

When connecting to CAS or SAS/Connect

As those certificates are always generated, you’ll need to retrieve the CA certificate and add it to your trustsore.

Run these commands from a linux server that does have access to your Kubernetes cluster. It must have the kubectl and the openssl command line tools installed.

kubectl -n $viya get secret sas-viya-ca-certificate-secret -o go-template='{{(index .data "ca.crt")}}' | base64 -d>ca_cert.pem

Follow the step described in the previous section to import this CA Certificate to your truststore.

Conclusion

Untrusted certificates can cause significant disruptions to the functionality and security of your SAS Viya platform, whether it's through browser warnings or internal communication failures. However, by diagnosing the issue with certificate chains and properly importing the required CA certificates into the appropriate trust stores, you can resolve these issues and restore secure communications.

As we continue this series on troubleshooting HTTPS certificate issues, stay tuned for our next post, where we will delve into another common challenge: mismatched domain names. This topic will further equip you with the knowledge needed to ensure a secure and efficient SAS Viya environment.

For a deeper understanding, I highly recommend the comprehensive 'Advanced Topics in Encryption on SAS Viya' training. This course provides a thorough exploration of encryption, covering everything from core concepts to advanced techniques. It also includes hands-on exercises and an associated lab environment, allowing you to apply what you've learned in real-world scenarios and further strengthen your knowledge of secure communication within SAS Viya.

Be sure to check out the complete series on Troubleshooting HTTPS Certificate Issues for your SAS Viya platform to guide you through various common challenges and their resolutions.

Find more articles from SAS Global Enablement and Learning here.