Troubleshooting HTTPS Certificate Issues in SAS Viya: Expired certificates

In the previous posts, we discussed the SSL process and the Kubernetes objects involved in managing SSL/TLS within a SAS Viya environment. Now, we turn our focus to one of the most common SSL issue: expired certificates.

Expired certificates can disrupt secure communication, leading to service outages and security risks. In this post, we’ll explore how to identify and troubleshoot expired certificates in SAS Viya. We'll walk through how to monitor expiration dates, renew and replace certificates, and prevent downtime by ensuring proper certificate management. Whether your certificates are issued by Cert-Manager, SAS OpenSSL, or manually provisioned, staying on top of expirations is key to keeping your platform secure and operational.

Note: The $viya environment variable is used on kubectl commands that require the --namespace flag. Make sure that the environment variable is created before you use these kubectl commands.

Expired Ingress Certificate

An expired certificate means that the certificate has exceeded its valid period. When this happens, browsers will display warnings to users, indicating that the site is not secure.

Symptoms

If the SSL certificate for your Ingress controller has expired, users will likely experience service outages or be presented with warnings that HTTPS is no longer secure.

Here is an example of the Error displayed if you were using Chrome:

Select any image to see a larger version.
Mobile users: To view the images, select the "Full" version at the bottom of the page.

Technical Diagnostic & Confirmation

To verify if your Ingress certificate has expired, run the following commands from a Linux machine with access to your Kubernetes cluster and ensure that kubectl and openssl are installed.

# Retrieve the Ingress host
INGRESS_HOST=$(kubectl -n $viya get $(kubectl -n $viya get configmap --output name | grep ingress-input) -o jsonpath='{.data.INGRESS_HOST}')
# Verify the certificate
echo X | openssl s_client -connect ${INGRESS_HOST}:443 -servername $INGRESS_HOST 2>/dev/null | openssl verify

If the certificate has expired, you will see output like:

stdin: C = US, ST = North Carolina, L = Cary, O = SAS, CN = fradae-p02205-rg.gelenable.sas.com
error 10 at 0 depth lookup:certificate has expired
OK

Resolution

The steps to renew the expired certificate depend on how it was provisioned:

If you were using the SAS provided OpenSSL Certificate Generator for the Ingress Certificate:

Delete the kubernetes secret holding the expired certificate
```
kubectl -n $viya delete secret sas-ingress-certificate
```

Delete the job “sas-create-openssl-ingress-certificate” (if still exists)

kubectl -n $viya delete job sas-create-openssl-ingress-certificate --ignore-not-found

Re-run the job (using yq) to recreate the Ingress certificate (minimum required)

cd ${deploy}
yq e '. | select(.kind == "Job" and .metadata.name =="sas-create-openssl-ingress-certificate")' site.yaml | kubectl -n $viya apply -f -

Or reapply you site.yaml manifest (be careful if you applied manual customizations after your last deployment, these will be reverted)

If you were using an IT provided certificate and key file for your ingress certificate:

Obtain the new ingress server certificate, ingress server key, and ingress server CA certificate.
Change the files in the site-config location that is referenced in the customer-provided-ingress-certificate.yaml file in the kustomization.yaml file.
Redeploy the SAS Viya platform software to apply the changes. (be careful if you applied manual customizations after your last deployment, these will be reverted)

If you were using the Cert Manager Certificate Generator for the Ingress Certificate:

Follow the steps outlined in the SAS® Help Center - Generate New Certificates or Renew Certificates When They Expire

Expired SAS Viya Services Certificates

Note: This only applies if your SAS Viya deployment is configured in Full-Stack TLS mode.

This scenario is uncommon, as SAS Viya service certificates are auto generated and regenerated whenever the service pods are restarted. The only case in which a service certificate might expire is if a pod hasn't been restarted for the full duration of the certificate's validity, which is set in the customer-provided-merge-sas-certframe-configmap.yaml ConfigMapGenerator.

Technical Diagnostic & Confirmation

Run these commands from a linux server that does have access to your Kubernetes cluster. It must have the kubectl and the openssl command line tools installed.

inCertificate='sas-logon-app-5579fdc5bf-6h7k5'
expiryDate=$(kubectl -n $viya get secret $inCertificate -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -enddate | awk -F'=' '{print $2}')
subject=$(kubectl -n $viya get secret $inCertificate -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -subject | awk -F'=' '{print $2}')
issuedBy=$(kubectl -n $viya get secret $inCertificate -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -issuer | awk -F'CN=' '{print $2}' | awk -F'/' '{print $1}')
SANs=$(kubectl -n $viya get secret $inCertificate -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509  -text -noout | grep -A1 "Subject Alternative Name" | tail -1)
echo -e "$inCertificate details are: \n" \
"  - It will expire on $expiryDate\n" \
"  - It was issued by $issuedBy\n" \
"  - For all Subject Alternative names $SANs"

You shoud see output like:

sas-logon-app-5579fdc5bf-rbblj details are:
- It will expire on Sep 18 16:34:28 2024 GMT
- It was issued by SAS Viya openssl Root CA Certificate
- For all Subject Alternative names                 DNS:aks-generic-24003689-vmss000005, DNS:fradae-p04210-rg.gelenable.sas.com, DNS:localhost, DNS:sas-logon-app, DNS:sas-logon-app-5579fdc5bf-rbblj, IP Address:10.0.153.132, IP Address:10.244.3.146, IP Address:127.0.0.1, IP Address:192.168.0.8

Resolution

Resolution is simple, restart the faulty pod or your SAS Viya platform completely.

Run these commands from a linux server that does have access to your Kubernetes cluster. It must have the kubectl command line tool installed.

currentTime=`date +%s`
kubectl -n $viya create job sas-stop-all-$currentTime --from cronjobs/sas-stop-all
kubectl -n $viya wait --timeout=7200s --for=condition=complete job/sas-stop-all-$currentTime
kubectl -n $viya create job sas-start-all-$currentTime --from cronjobs/sas-start-all

Expired SAS Viya Services Root CA used by the generators

As I said previously, the SAS Viya Services Certificates are always generated.

When using Cert-Manager, without any customized configurations, the sas-viya-ca-certificate secret holds the Root CA certificate and has a default duration of 5 years and should automatically be renewed 1h prior the expiration (documented under the cert-manager-issuer overlay folder). Remember, you will have to restart your SAS Viya platform after a Root CA certificate is renewed.

Technical Diagnostic & Confirmation

Run these commands from a linux server that does have access to your Kubernetes cluster. It must have the kubectl and the openssl command line tools installed.

inCertificate='sas-viya-ca-certificate-secret'
expiryDate=$(kubectl -n $viya get secret $inCertificate -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -enddate | awk -F'=' '{print $2}')
subject=$(kubectl -n $viya get secret $inCertificate -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -subject | awk -F'=' '{print $2}')
issuedBy=$(kubectl -n $viya get secret $inCertificate -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -noout -issuer | awk -F'CN=' '{print $2}' | awk -F'/' '{print $1}')
SANs=$(kubectl -n $viya get secret $inCertificate -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509  -text -noout | grep -A1 "Subject Alternative Name" | tail -1)
echo -e "$inCertificate details are: \n" \
"  - It will expire on $expiryDate\n" \
"  - It was issued by $issuedBy\n" \
"  - For all Subject Alternative names $SANs"

You shoud see output like:

sas-viya-ca-certificate-secret details are:
- It will expire on Sep 18 15:32:54 2029 GMT
- It was issued by SAS Viya openssl Root CA Certificate
- For all Subject Alternative names

Resolution

When using SAS OpenSSL generator, there is no automatic process to renew the Root CA certificate after the 5 years used during the initial deployment. As on each service pods restart, its presence is checked, you only need to delete the expired Root CA certificate and restart your SAS Viya platform.

Run these commands from a linux server that does have access to your Kubernetes cluster. It must have the kubectl command line tool installed.

kubectl -n $viya delete secret sas-viya-ca-certificate-secret
currentTime=`date +%s`
kubectl -n $viya create job sas-stop-all-$currentTime --from cronjobs/sas-stop-all
kubectl -n $viya wait --timeout=7200s --for=condition=complete job/sas-stop-all-$currentTime
kubectl -n $viya create job sas-start-all-$currentTime --from cronjobs/sas-start-all

Conclusion

In this post, we explored the critical issue of expired certificates within your SAS Viya platform. By understanding the symptoms of expired certificates and following the diagnostic steps outlined, you can quickly identify and resolve these issues to maintain secure communication across your services. Whether you are using the SAS OpenSSL generator, Cert-Manager, or manually provisioned certificates, proactive management of expiration dates is essential to prevent service disruptions and security risks.

As we continue this series on troubleshooting HTTPS certificate issues, stay tuned for our next post, where we will delve into another common challenge: untrusted certificates. This topic will further equip you with the knowledge needed to ensure a secure and efficient SAS Viya environment.

For a deeper understanding, I highly recommend the comprehensive 'Advanced Topics in Encryption on SAS Viya' training. This course provides a thorough exploration of encryption, covering everything from core concepts to advanced techniques. It also includes hands-on exercises and an associated lab environment, allowing you to apply what you've learned in real-world scenarios and further strengthen your knowledge of secure communication within SAS Viya.

Be sure to check out the complete series on Troubleshooting HTTPS Certificate Issues for your SAS Viya platform to guide you through various common challenges and their resolutions.

Find more articles from SAS Global Enablement and Learning here.