How to survive SAS Viya self-signed certificates expiration
Do you have a SAS Viya environment that was installed some months ago? Once it gets to its first birthday, you may find that, like a real kid, along with many joys it can also bring unexpected problems.
Important update for SAS Viya 3.5
The original version of this blog post was written for SAS Viya 3.4.
SAS Viya 3.5 creates certificates that are valid for 7 years by default, so this should not happen anymore. It can still happen if your 3.5 environment is an upgrade in place of an older one, installed with 3.4 or even 3.3. In that case, the upgrade did not change the original Apache certificate, which can still be only valid for 1 year.
In any case, both for new installations and upgrades, the SAS Viya 3.5 version of the /opt/sas/viya/home/bin/replace_httpd_default_cert.sh script referenced below uses a new syntax and is already set to create a 7-year long certificate. If you encounter this issue with SAS Viya 3.5, you can still follow the instructions in the paragraph “The way out” but only follow steps 2 to 5 – i.e., skip steps 1 and 6.
Some background
SAS Viya is a mature component of the SAS platform. Many customers have deployed in production environments solutions based on SAS Viya, and countless are the images maintained by different SAS business units for internal employees. As part of the ongoing effort to meet and exceed industry-standard requirements, starting with SAS Viya 3.3, all deployments default to encrypted communications, thanks to the use of standard TLS enabled on all service endpoints.
One of the key principles to make this possible is the deployment of a frontend Apache HTTP server, used as a proxy by SAS Viya servers and services.
During SAS Viya deployment, the installation of the sas-httpproxy package triggers a dependency on both Apache HTTP Server (httpd) and mod_ssl. Httpd and mod_ssl are automatically installed by YUM, and the installation of mod_ssl creates a private key and self-signed certificate. In other words, initially, a self-signed certificate is created by Apache itself, not by SAS. In a subsequent phase of the SAS Viya deployment, this self-signed certificate is updated by SAS Viya playbooks. This happens because the certificate contains, in the common name field (CN), the hostname of the machine at the point in time that mod_ssl is installed, but does not include any Subject Alternate Names (SAN). SAS Viya deployment modifies the original certificate to add additional entries in the SAN field, but leaves other properties, including the expiration date, to their default value. By default, Apache self-signed certificates are valid for one year since their creation.
It’s easy to understand that, with many SAS Viya deployments performed in 2018, after the release of version 3.3, some of them are now past that default expiration date.
So, what happens?
Once a TLS certificate has expired, it is rejected by clients attempting to communicate with any server that uses it, and the connection fails. This is good, because it prevents fraudulent use of old certificates! If you have the good habit of always using https, you can notice that this is reported by the browser with a message:
Browser Warnings
Left: Microsoft Edge - Right: Google Chrome
On the contrary, if you connect to the SAS Viya environment using the “unsecure” http protocol, you may not notice any issue and successfully log in. Yet, internal connections between services, which always happen through secure channels, still fail. Eventually, you may experience failure messages, for example in the following areas:
- Issues using SAS Studio V (unable to start a compute server)
Select any image to see a larger version.
Mobile users: To view the images, select the "Full" version at the bottom of the page. - Issues running models in Model Studio (unable to start a compute server)
- Errors in CAS (unable to authenticate to SASLogon)
An example of error messages from CAS:
2019-01-20T00:00:16,508 ERROR [00003526] cas local MAIN NoUser [sslopenssl2.c:6453] - OpenSSL error 336134278 (0x14090086) occurred in SSL_connect/accept at line 4717, the error message is "error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed". 2019-01-20T00:00:16,508 ERROR [00003526] cas local MAIN NoUser [skstssl.c:1248] - Secure communications error status 807ff008 description "192.168.1.1: The certificate sent from the remote host cannot be validated by any of the public keys in the root certificate store specified by SSLCALISTLOC or SSLCACERTDIR." 2019-01-20T00:00:16,509 ERROR [00003526] cas local MAIN NoUser [tkjwk.c:1325] - getKeySet: Could not send request to server. 2019-01-20T00:00:16,509 ERROR [00003526] cas local MAIN NoUser [tkjwk.c:1326] - Encryption run-time execution error - Unable to authenticate with the sas-admin CLI (unable to authenticate to SASLogon)
- ...
A possible cause of confusion is that administrators can still successfully connect to SAS Environment Manager via http and the dashboard will not report any issue. They would have to dig into logs to spot any error messages.
To confirm this is the actual cause of end-users' problems, a quick way to check the certificate expiration date is by using the command:
openssl x509 -noout -in /etc/pki/tls/certs/localhost.crt -enddate
The best practice
Well, in an ideal world, all this should not happen. In fact, in the official documentation, we recommend our customers that they should not use the default self-signed certificate, but rather add their own.
For reference, the Transport Layer Security section of the SAS Viya 3.4 for Linux: Deployment Guide reads:
"If you do not add compliant certificates and instead keep the default security settings and certificates, end users will see a standard web browser warning message. SAS recommends replacing the certificates before giving end users access to the software."
Obviously, this also means that, after you install your own certificates, you ensure that they are valid and replace them before the expiration.
Since we do not live in an ideal world, somebody may either choose to not add their own certificate or simply forget about this - even worse, they may not read that section of the documentation at all! In all these cases, for the first year, connections to SAS web applications are successful - though with the noted web browser warning messages. End users could choose to import the self-signed certificate into their browser trust store, thus eliminating any warning messages even when using https. All works fine ... until the 1-year mark expires.
The way out
There is a workaround to recover from the situation where the self-signed certificate expires. The certificate can be manually reissued - either for another year, by default, or for the number of days the user desires. It can then be redistributed across the deployment, which will return to a functioning state.
There is no need to wait until the certificate expiration – this can be done as soon as the initial deployment has completed.
- Modify /opt/sas/viya/home/bin/replace_httpd_default_cert.sh to add --days as in the highlighted row
... # Self-sign CSR ${SASCRYPTOHOME}/sas-crypto-management \ selfsign \ --days 2557 \ --csr ${DEFAULTCSRPATH}/localhost.csr \ --out-file ${DEFAULTCERTPATH}/localhost.crt \ --out-form pem \ --signing-key ${DEFAULTKEYPATH}/localhost.key ...In this example, the expiration is bumped up to 7 years (to match the other long-term certificates that are created for other SAS Viya components).
- Execute the script as root with --force option:
sudo /opt/sas/viya/home/bin/replace_httpd_default_cert.sh --force 2019-06-18 20:21:20.049 INFO [Log.go:41] [sas-crypto-management-command] - {"0":"[]"} gencsr.go:132 succesfully created CSR! 2019-06-18 20:21:20.085 INFO [Log.go:41] [sas-crypto-management-command] - {"0":"[]"} selfsign.go:282 successfully created X.509 template for cert generation 2019-06-18 20:21:20.087 INFO [Log.go:41] [sas-crypto-management-command] - {"0":"[]"} selfsign.go:284 signing certificate... 2019-06-18 20:21:20.090 INFO [Log.go:41] [sas-crypto-management-command] - {"0":"[]"} selfsign.go:292 succesfully signed certificate! - Restart the httpproxy service:
sudo service sas-viya-httpproxy-default restart - Re-distribute the HTTPD certificate to the whole environment:
cd /sas/install/sas_viya_playbook ansible-playbook -i inventory.ini utility/distribute-httpd-certs.yml - Restart all services:
sudo /etc/init.d/sas-viya-all-services stop sudo /etc/init.d/sas-viya-all-services start - Remove the expired certificates from all truststores. This is a manual step. Without it, the old expired certificate may still be registered in the truststores and be used by client software to verify the https connection, even when the server presents the new certificate.
- On the server: edit the truststore with a text editor (remember to make a backup copy before editing!):
sudo vi /opt/sas/viya/config/etc/SASSecurityCertificateFramework/cacerts/trustedcerts.pemfind the old certificate and delete it. The new one should already be there.
- On the client: If the self-signed certificate was imported into the client truststore (such a Window’s), remove it and import the new one.
- On the server: edit the truststore with a text editor (remember to make a backup copy before editing!):
Does this really apply to me?
Maybe you just installed a brand new SAS Viya environment, after removing an older one from the same hosts. Since you undeployed and re-deployed, everything should be new. You think you should not encounter this problem for at least another year... Well, unfortunately, no.
This issue is still present in different scenarios:
- SAS Viya is updated with a hotfix or a newer version of a component.
- A new product is added to the original installation.
- SAS Viya is upgraded to a newer release (for example, from SAS Viya 3.3 to SAS Viya 3.4).
- SAS Viya is uninstalled, then the same host is used for a brand-new installation (no matter what release).
- Everyone only connects to web applications using http instead of https.
In all of the above cases, the Apache server keeps using the certificate created with the initial deployment, no matter of the subsequent changes. At the 1-year mark since that initial deployment, SAS Viya will start complaining.
Improvements are coming here
If you install a recent release of SAS Viya ( starting with SAS Viya 3.4 Revision 19w30, released in July 2019 ) you will notice that the change to /opt/sas/viya/home/bin/replace_httpd_default_cert.sh described in step #1 above is not required anymore: it is already there. Any new deployment, with a recent SAS Viya release, on a new set of hardware, will incorporate the change you could be all set for the next 7 years.
Still, let me close with a word of caution. This fix should still not replace good security and administration habits: as it is suggested in the Configure and Update TLS and HTTPS section of the SAS Viya 3.4 Administration Guide: "SAS recommends that you enhance the security by replacing this certificate with a custom certificate that is generated according to the security standards at your enterprise."
I have read article completely and applied everything written there in my Viya 3.5 RACE image. I still can not log in to web applications, because LDAPS certificate expired.
And because LDAP server is in docker, looks like ansible does not propagate new certificates everywhere...
