SAS Viya 2020.1 (and later) Front Door Ingress HTTPS with Customer Provided Certificate
A common scenario for many customers is for their IT team to provide the TLS certificate they want to use to secure traffic to their environments. In this blog, we will look at the steps that need to be completed to secure access into SAS Viya 2020.1 (and later) with such a certificate. We will examine two different scenarios; in the first scenario, the customer just wants to update the standard "Secure by default" configuration to use their certificate for the Ingress Controller (NGINX). In the second scenario the customer is happy with the security options provided by Azure within their Kubernetes cluster and only want HTTPS to the Ingress Controller (NGINX).
Scenario 1: Front Door HTTPS with Customer Provided Certificate
In this scenario the customer is only looking to alter the default configuration we discussed before to change the certificate pair (private key and signed server identity certificate) used by the Ingress Controller. This means that the customer provided certificate pair (private key and signed server identity certificate) will be used for the external HTTPS connections into the environment, while the SAS configured Certificate Authority is used inside the environment, as shown below. Traffic from outside the Kubernetes cluster will be decrypted by the Ingress Controller (NGINX) using the provided certificate pair (private key and signed server identity certificate) and then re-encrypted for the network connection from the Ingress Controller to the SAS Viya services.
Select any image to see a larger version.
Mobile users: To view the images, select the "Full" version at the bottom of the page.
Scenario 2: Only Front Door HTTPS with Customer Provided Certificate
In this scenario the customer has certificate pair (private key and signed server identity certificate) that needs to be applied to the Ingress Controller for their SAS Viya 2020.1 (and later) environment. This will enable HTTPS for connections from the end-user browser into the environment. Unlike the first scenario the customer is happy with the network security provided by their Kubernetes provider and wants to run the internal communication without additional encryption, as shown below. So, the Ingress Controller (NGINX) still acts as the HTTPS endpoint for external connections, but now there is no re-encryption of network traffic within the Kubernetes cluster.
Prerequisites for both scenarios
The customer will provide their TLS certificate and associated private key, any Intermediate Certificate Authority (CA) certificates and the root CA certificate. They are only likely to provide the root CA certificate if the CA is internal to their organization. However, for a commercial CA this is not necessary since the root CA will be included in the Mozilla bundle of CA certificates. You can confirm if a CA is included in the Mozilla bundle here: https://ccadb-public.secure.force.com/mozilla/IncludedCACertificateReport.
If the files provided by the customer are not in the correct format or are bundled into a container file such as a *.pfx, OpenSSL can be used to manipulate them. For example, if the customer provides a bundled container file certname.pfx you can use the following OpenSSL commands:
# extract private key openssl pkcs12 -in certname.pfx -nocerts -out enckey.pem -nodes # extract all certificates openssl pkcs12 -in certname.pfx -nokeys -out chain.pem # remove passphrase from private key openssl rsa -in enckey.pem -out server.key
You can then examine the chain.pem file with a text editor to confirm it only contains the sever TLS certificate and any intermediate CA certificates. If the root CA certificate is included in the chain.pem this can be extracted, using a text editor, into a separate ca.pem file.
You can use OpenSSL to read the certificate content, for example:
openssl x509 -text -noout -in chain.pem |grep 'Subject:\|Issuer:'
Since the chain.pem contains multiple certificates this command will only output the first certificate in the file, and it will display both the Subject and Issuer fields. If the Subject and Issuer fields are different this certificate is either the server certificate or an intermediate certificate. Only a root CA certificate will have the same text for both Subject and Issuer.
You can automatically split the chain.pem file into separate files using the following command:
csplit -z -f cert_ chain.pem /END/1 '{*}'
This will result in several files called cert_00, cert_01, etc. You can then use the following command to look at the Subject and Issuer for each file:
for i in cert_*; do echo $i; openssl x509 -text -noout -in $i|grep 'Subject:\|Issuer:';done
For example, this might output:
cert_00
Issuer: DC=com, DC=SAS, CN=SAS SHA2 Issuing CA01
Subject: C=US, ST=North Carolina, L=Cary, O=SAS Institute Inc., OU=GEL, CN=dailymirror.rext##-###.race.sas.com/emailAddress=some.name@sas.com
cert_01
Issuer: DC=com, DC=sas, CN=SAS SHA2 Root CA
Subject: DC=com, DC=sas, CN=SAS SHA2 Root CA
cert_02
Issuer: DC=com, DC=sas, CN=SAS SHA2 Root CA
Subject: DC=com, DC=SAS, CN=SAS SHA2 Issuing CA01
This shows us that cert_00 is the server certificate, cert_02 is the intermediate CA, and cert_01 is the root CA. We can then combine the server certificate and intermediate CA, rename the root CA and server private key:
cat cert_00 > sas-NGINX-ingress.crt; cat cert_02 >> sas-NGINX-ingress.crt; cat cert_01 > ca.crt; cat server.key > sas-NGINX-ingress.key
You want to end up with either two or three files:
- Server and intermediate CA certificates - sas-NGINX-ingress.crt
- Server private key - sas-NGINX-ingress.key
- (Optional) Root CA certificate - ca.crt
These steps can be completed on any machine with OpenSSL installed, this does not need to be part of the SAS Viya environment. The files can then be securely copied to the host where the kubectl commands will be run from.
Once you have the required files in the correct format these can be used in the environment. The documentation and the included readme recommend creating a sub-folder of site-config called security to hold the customized content you will include. You should copy the example sas-bases/examples/security/customer-provided-ingress-certificate.yaml to this site-config/security directory. You will place the TLS certificate with any Intermediate CA certificates, the private key, and possibly the root CA certificate place in the site-config/security directory.
Update the site-config/security/customer-provided-ingress-certificate.yaml file to set the tls.crt, and tls.key. For example, this might look like the following:
--- apiVersion: builtin kind: SecretGenerator metadata: name: sas-ingress-certificate files: - tls.crt=site-config/security/sas-NGINX-ingress.crt - tls.key=site-config/security/sas-NGINX-ingress.key type: "kubernetes.io/tls"
Next, if you need to provide the Root CA certificate you should follow the documented steps for including additional CA certificates. This means copying sas-bases/examples/security/customer-provided-ca-certificates.yaml to your site-config/security directory. Then update the customer-provided-ca-certificates.yaml file to point to your ca.crt file. For example, this might look like the following:
--- apiVersion: builtin kind: ConfigMapGenerator metadata: name: sas-customer-provided-ca-certificates behavior: merge files: - site-config/security/cacerts/ca.crt
Once we have the customer-provided-ingress-certificate.yaml and possibly customer-provided-ca-certificates.yaml files we can look to implement either of the two scenarios.
Implement Scenario 1
In this scenario we will have the majority of the default documented content in the kustomization.yaml. Under resources, for TLS we have:
- sas-bases/overlays/cert-manager-issuer - sas-bases/overlays/network/ingress/security
Under transformers, but before the sas-bases/overlays/required/transformers.yaml line we have we have:
- sas-bases/overlays/network/ingress/security/transformers/product-tls-transformers.yaml - sas-bases/overlays/network/ingress/security/transformers/ingress-tls-transformers.yaml - sas-bases/overlays/network/ingress/security/transformers/backend-tls-transformers.yaml
But note we do not include the following under transformers:
- site-config/security/cert-manager-provided-ingress-certificate.yaml
Now to specify the customer-provided-ingress-certificate.yaml and possibly the customer-provided-ca-certificates.yaml files we created earlier we add to the bottom of the kustomization.yaml, or beneath any existing content under generators:
generators: - site-config/security/customer-provided-ingress-certificate.yaml # configures ingress to use a secret which contains customer provided certificate and key - site-config/security/customer-provided-ca-certificates.yaml
Don’t forget to ensure the SAS_SERVICE_URL specifies HTTPS instead of HTTP.
You can then build the updated site.yaml and deploy SAS Viya as per the documentation:
NS={{ namespace }}
kustomize build -o site.yaml
kubectl apply -n ${NS} --selector="sas.com/admin=cluster-wide" -f site.yaml
kubectl wait -n ${NS} --for condition=established --timeout=60s -l "sas.com/admin=cluster-wide" crd
kubectl apply -n ${NS} --selector="sas.com/admin=cluster-local" -f site.yaml --prune
kubectl apply -n ${NS} --selector="sas.com/admin=namespace" -f site.yaml --prune
Result of Scenario 1
Once the site.yaml has been applied the Ingress Controller (NGINX) will now use the TLS certificate and private key provided by the customer. This will provide encryption on connections into the environment. This certificate pair (private key and signed server identity certificate) is not used provide encryption within the environment; the standard SAS generated self-signed CA certificate is used on the connections within the environment.
However, it is useful to understand what actually happens when the site.yaml is applied. The following diagrams illustrates this:
At the top of the diagram we show the customer-provided-ingress-certificate.yaml, possibly the customer-provided-ca-certificates.yaml files, the default sas-bases/overlays/cert-manager-issuer/resources.yaml file, and the updates that are made to the kustomization.yaml. Then at the bottom of the diagram we illustrate what happens:
- The internal SAS provided CA certificate is generated and the two cert-manager issuers are created, shown in the green box. This is defined by the sas-bases/overlays/cert-manager-issuer/resources.yaml file also shown in green.
- A Kubernetes configMap is created from the customer-provided-ca-certificates.yaml file containing additional CA certificates called sas-customer-provided-ca-certificates
- A Kubernetes secret is created containing the tls.key and tls.cert contents from the customer-provided-ingress-certificate.yaml file named sas-ingress-certificate-##########
- A Kubernetes configMap is created referencing the name of the Kubernetes secret holding the Ingress Controller certificate
- A Kubernetes configMap is created defining the location of the truststores for use by the different SAS services
- The Ingress definitions for the SAS services are updated to include a reference to the Kubernetes secret holding the Ingress Controller certificate
- The updates to the Ingress definitions trigger an update of the nginx.conf to include the private key and certificate information
- An init container sas-certframe is now included in each SAS pod, shown in the blue box at the bottom of the diagram
- The sas-certframe init container reads information from the referenced configMaps
- The sas-certframe init container loads the tls.key and tls.cert referenced by the Ingress Controller certificate Kubernetes secret
- The sas-certframe init container loads the additional CA certificates from the Kubernetes configMap
- The sas-certframe init container requests a certificate pair from the cert-manager issuer: sas-viya-issuer
- Cert-Manager places the resulting certificate pair for the service in a Kubernetes secret
- The sas-certframe init container loads the service certificate pair from the Kubernetes secret once the certificate has a status of "Ready"
- The sas-certframe init container then builds the truststores for use by the different SAS services running in the pod based on the Ingress Controller certificate, SAS internal CA and additional CA certificates which is then saved to the /security mount point
- The sas-certframe init container also makes the service certificate pair available in the correct format and saves these to the /security mount point
- The SAS service container loads the environment variables providing the location of the truststores and certificate pair from the Kubernetes configMaps
- The SAS service mounts the truststores and certificate pair built by the sas-certframe init container
This results in the services now being available over HTTPS and the truststores used by the services including the customer provided Ingress Controller signed identity certificate.
Implement Scenario 2
The key part of scenario 2 is that there will be no encryption within the Kubernetes cluster. HTTPS will be used up to the Ingress Controller (NGINX) and then all traffic behind the Ingress Controller will be unencrypted. Since this is such a departure from the secured by default stance we will need to remove more items from the default documented kustomization.yaml. Under resources, for front-door HTTPS we only have:
- sas-bases/overlays/network/ingress/security
Under transformers, but before the sas-bases/overlays/required/transformers.yaml line we have:
- sas-bases/overlays/network/ingress/security/transformers/truststore-transformers-without-backend-tls.yaml - sas-bases/overlays/network/ingress/security/transformers/ingress-tls-transformers.yaml
But note we do not include the following under transformers:
- sas-bases/overlays/network/ingress/security/transformers/product-tls-transformers.yaml - sas-bases/overlays/network/ingress/security/transformers/backend-tls-transformers.yaml - site-config/security/cert-manager-provided-ingress-certificate.yaml
Now to specify the customer-provided-ingress-certificate.yaml and possibly the customer-provided-ca-certificates.yaml files we created earlier we add to the bottom of the kustomization.yaml, or beneath any existing content under generators:
generators: - site-config/security/customer-provided-ingress-certificate.yaml # configures ingress to use a secret which contains customer provided certificate and key - site-config/security/customer-provided-ca-certificates.yaml
Don’t forget to ensure the SAS_SERVICE_URL specifies HTTPS instead of HTTP.
You can then build the updated site.yaml and deploy SAS Viya as per the documentation:
NS={{ namespace }}
kustomize build -o site.yaml
kubectl apply -n ${NS} --selector="sas.com/admin=cluster-wide" -f site.yaml
kubectl wait -n ${NS} --for condition=established --timeout=60s -l "sas.com/admin=cluster-wide" crd
kubectl apply -n ${NS} --selector="sas.com/admin=cluster-local" -f site.yaml --prune
kubectl apply -n ${NS} --selector="sas.com/admin=namespace" -f site.yaml --prune
Result of Scenario 2
Once the site.yaml has been applied the Ingress Controller (NGINX) will now use the certificate pair (private key and signed server identity certificate) provided by the customer. This will provide encryption on connections into the Ingress Controller. However, no encryption will be used beyond the Ingress Controller.
However, it is useful to understand what actually happens when the site.yaml is applied. The following diagrams illustrates this:
At the top of the diagram we show the customer-provided-ingress-certificate.yaml, possibly the customer-provided-ca-certificates.yaml files and the updates that are made to the kustomization.yaml. Then at the bottom of the diagram we illustrate what happens:
- A Kubernetes secret is created containing the tls.key and tls.cert contents from the customer-provided-ingress-certificate.yaml file called sas-ingress-certificate-##########
- A Kubernetes configMap is created from the customer-provided-ca-certificates.yaml file containing additional CA certificates called sas-customer-provided-ca-certificates
- A Kubernetes configMap is created referencing the name of the Kubernetes secret holding the Ingress Controller certificate
- A Kubernetes configMap is created defining the location of the truststores for use by the different SAS services
- The Ingress definitions for the SAS services are updated to include a reference to the Kubernetes secret holding the Ingress Controller certificate
- The updates to the Ingress definitions trigger an update of the nginx.conf to include the private key and certificate information
- An init container sas-certframe is now included in each SAS pod
- The sas-certframe init container reads information from the two referenced configMaps, loads the tls.key and tls.cert referenced by the Kubernetes secret, and loads additional CA certificates from the configMap
- The sas-certframe init container then builds the truststores for use by the different SAS services running in the pod based on the Ingress Controller certificate and any additional CA certificates which is then saved to the /security mount point
- The SAS service container loads the environment variables providing the location of the truststores from the Kubernetes configMap and mounts the truststores built by the sas-certframe init container
This results in the services now being available over HTTPS and the truststores used by the services including the customer provided certificate pair (private key and signed server identity certificate).
Closing Remarks
In this blog we have presented two of the most common scenarios for HTTPS setup with SAS Viya 2020.1 (and later). The first where all network traffic is still encrypted but the Ingress Controller (NGINX) now uses the customer provided certificate pair (private key and signed server identity certificate). Then the second scenario where only the Ingress Controller (NGINX) is configured for encryption using the customer provided certificate pair (private key and signed server identity certificate) and where all other network traffic is unencrypted.
With these scenarios we have only been concerned with the HTTPS front-door to the SAS Viya environment. We have not addressed the direct connections to CAS and SAS Connect Spawner. We will look at these in future blog posts.
Note: This article addresses features of SAS Viya 2020.1 (and later)
