Troubleshooting HTTPS Certificate Issues in SAS Viya: Analyzing Logs

1 Like

In our previous posts, we have explored various aspects of HTTPS certificate issues in your SAS Viya platform, including Expired Certificates, Mismatched Domain Names, and Incomplete Certificate Chain. In this final post of the series, we will focus on troubleshooting logs, specifically those from the sas-certframe init container and the Ingress controller. These logs provide essential insights into SSL-related issues, helping you diagnose and resolve problems such as missing certificates, truststore errors, and Ingress misconfigurations.

Note: The $viya environment variable is used on kubectl commands that require the --namespace flag. Make sure that the environment variable is created before you use these kubectl commands.

Diagnosing Issues in the sas-certframe Init Container

The sas-certframe init container is responsible for managing "internal" SSL certificates in SAS Viya. If this container fails, pods may get stuck in an Init or CrashLoopBackOff state. Checking its logs can help diagnose issues such as missing certificates, failed updates, or truststore errors.

To retrieve the logs, run:

POD_NAME=$(kubectl -n $viya get pods -l app=sas-logon-app -o jsonpath='{.items[0].metadata.name}')
kubectl -n $viya logs $POD_NAME -c sas-certframe

A successful execution of the container typically follows these steps:

Initialization – The container starts and confirms its version (e.g., 3.62.0).
Certificate Verification & Generation – It checks for an existing CA certificate (sas-viya-ca-certificate-secret). If missing and using OpenSSL, a new certificate will be created.
Certificate Storage – The container saves the generated certificates in /security, this location will be shared with the service container.
Truststore Setup – The truststore is updated with Mozilla CA certificates, the Ingress CA chain, the SAS CA chain, and any additional customer-provided CA certificates also stored in /security
Completion – The logs confirm successful execution.

Common Errors and Resolutions

Truststore Error:

If the logs contain messages like:

error parsing certificate: x509: failed to parse crypto resource fullFileName:/customer-provided-ca-certificates/additional_ca.pem

This indicates a formatting issue with the additionnal, customer provided CA certificate. To resolve:

Ensure the certificate is in PEM format and correctly formatted.
Check for missing BEGIN CERTIFICATE and END CERTIFICATE lines.

Run this command to verify its validity.

openssl x509 -in additional_ca.pem -text -noout

Missing Ingress Certificate Secret:

If you see:

failed to get secret: secrets "sas-ingress-certificate" not found

It means the Ingress certificate secret is missing, preventing the truststore from updating. Check if the secret exists using:

kubectl -n $viya get secret sas-ingress-certificate

If it's missing, review your TLS configuration and regenerate the secret if necessary.

CertmanagerCACertGenerator.EnsureCACertificateExists (when using cert-manager)

If you see:

CA certificate Secret in CaCertSecretName does not exist. Ensure cert-manager Issuer and CA Certificate resources have been created. [CaCertSecretName:sas-viya-ca-certificate-secret]

It means the SAS Viya CA certificate secret (sas-viya-ca-certificate-secret) is missing, implying the cert-manager component has not been able to create it. Check if the secret exists using:

kubectl -n $viya get secret sas-viya-ca-certificate-secret

If the secret is missing, ensure that the cert-manager Issuer and CA Certificate resources are properly defined and applied:
```
kubectl -n $viya get issuers -o wide
```

By examining the logs and understanding whether SAS Viya is using cert-manager or OpenSSL, you can efficiently troubleshoot SSL certificate issues and ensure a smooth certificate management process.

Diagnosing Issues in the Ingress Controller logs

The Ingress controller manages external traffic to your SAS Viya platform and handles SSL termination. If there are issues with TLS certificates, you may encounter SSL handshake failures, expired or incorrect certificates, or misconfigured Ingress rules. Analyzing the Ingress controller logs can help identify and resolve these problems.

How are TLS certificates loaded?

Loading the Default TLS Certificate:
- When the Ingress controller starts, it loads the default TLS certificate.
- If no TLS certificate is specified in an Ingress resource, the default certificate is used.
- This is identified in the logs by the message: "SSL fake certificate created" file="/etc/ingress-controller/ssl/default-fake-certificate.pem".
- If you are accessing using an IP address (instead of a valid Ingress name) or a alternative hostname, issues with Ingress rules or misconfigurations (e.g., SAN DNS) may occur.
Select any image to see a larger version.
Mobile users: To view the images, select the "Full" version at the bottom of the page.
Loading Secrets for Ingress Objects:
- When an Ingress resource (Kubernetes object) specifies a TLS section, it references a Kubernetes secret that contains the certificate and private key.
- The Ingress controller monitors these secrets for changes and reloads them when updated.
- If the referenced secret is missing, misconfigured, or contains an expired certificate, SSL errors will arise.

Diagnosing issues

To inspect the logs of the Ingress controller and check for SSL-related errors, use the following commands:

POD_NAME=$(kubectl -n $ingress get pods -l app.kubernetes.io/instance=ingress-nginx -o jsonpath='{.items[0].metadata.name}')
kubectl -n $ingress logs $POD_NAME --tail=-1

Look for log messages that indicate problems with certificate validation, secret loading, or Ingress configuration. Common log entries include:

certificate is not valid (expired or not yet valid)
no SSL certificate provided for this server name (indicating a missing or misconfigured secret)
failed to find secret referenced in Ingress resource

Resolving certificate issues

Ensure Proper References in Ingress Resources: Ensure that each Ingress resource with TLS enabled references an existing Kubernetes secret:
```
kubectl -n $viya get secret $(kubectl -n $viya get ingress  -o jsonpath="{.spec.tls[*].secretName}")
```
Note: You can use sas-logon-app (as an example) for your ingress name

Check Secrets for Valid Certificates: Inspect the secrets to ensure they contain valid certificates and keys:

kubectl -n $viya describe secret $(kubectl -n $viya get ingress  -o jsonpath="{.spec.tls[*].secretName}")
kubectl -n $viya get secret $(kubectl -n  get ingress  -o jsonpath="{.spec.tls[*].secretName}") -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -text

Reload the Ingress Controller: If you make changes to the certificates, restart the Ingress controller to ensure it reloads the updated secrets:
```
kubectl -n $ingress rollout restart deployment nginx-ingress-controller
```

By following these steps, you can ensure that your Ingress controller loads and applies the correct TLS certificates, preventing SSL handshake and routing issues.

Conclusion

In this post, we examined how to troubleshoot SSL-related issues by analyzing the logs of the sas-certframe init container and the Ingress controller. We covered common errors, diagnostic steps, and resolutions to ensure proper certificate management and secure communication within your SAS Viya platform.

This concludes our series on troubleshooting HTTPS certificate issues. We hope this guide has provided valuable insights and practical solutions for managing SSL configurations effectively.

For a deeper understanding, I highly recommend the comprehensive 'Advanced Topics in Encryption on SAS Viya' training. This course provides a thorough exploration of encryption, covering everything from core concepts to advanced techniques. It also includes hands-on exercises and an associated lab environment, allowing you to apply what you've learned in real-world scenarios and further strengthen your knowledge of secure communication within SAS Viya.

Be sure to check out the complete series on Troubleshooting HTTPS Certificate Issues for your SAS Viya platform to guide you through various common challenges and their resolutions.

Find more articles from SAS Global Enablement and Learning here.