SAS Viya 3.3 Managing Trust Stores

In this post, continuing from the previous article where we looked at an overview of encryption in-motion, I want to look at managing the trust stores with SAS Viya 3.3. Ensuring the trust stores are updated with any additional certificates is critical to a correctly operating environment. We will look at the mechanisms you can use to manage the trust stores and recommend what approach you should take.

With SAS Viya 3.3 the SAS Security Certificate Framework provides two trust stores, one in Base64 PEM encoded format and one in a Java keystore format. These two files have the same content. The two files are necessary for the different types of technology deployed in the SAS Viya 3.3 environment. The files are the trustedcerts.pem and trustedcerts.jks and located in:

/opt/sas/viya/config/etc/SASSecurityCertificateFramework/cacerts/

These contain the Mozilla bundle of trusted Certificate Authority (CA) certificates and other certificates added as part of the deployment process.

The Ansible deployment process for SAS Viya 3.3 automatically obtains the HTTPS certificate from the Apache HTTP Server and adds this to the SAS Configuration Server under the key-value store "cacerts". Ansible uses the value of HTTPD_CERT_PATH to create an additional file in the SAS Security Certificate Framework under the cacerts directory, listed above. The deployment process then builds the trusted certs files using the Mozilla bundle and these two items. The trusted certs files are distributed across all the hosts in the Ansible inventory.

Adding to the Trust Stores

To add a new CA certificate to the trust stores, for example to enable LDAPS connections, you should complete the following:

Add the Base64 PEM encoded certificate to the sitedefault.yml file in:

/opt/sas/viya/config/etc/consul.d/default

For example, to add your own internal CA certificate you would add the following:

cacerts:
    MyRootCA: |
        -----BEGIN CERTIFICATE-----
        MIIFZTCCA02gAwIBAgIQIbEH2kWtB5NBMg9WdJc0jzANBgkqhkiG9w0BAQsFADBF
        MRMwEQYKCZImiZPyLGQBGRYDY29tMRMwEQYKCZImiZPyLGQBGRYDc2FzMRkwFwYD
        ...
        nhe0LRWZYEHeq9xxo5rHH/vcqB+nv2pIwqRbugyw80azdurVKn6cxMI=
        -----END CERTIFICATE-----

Restart the SAS Configuration Server to pick-up the changes to the sitedefault.yml file. Remember only new key-value pairs are added, existing key-value pairs will not be updated.
Use the Ansible playbook to rebuild the trust stores across all the hosts in the SAS Viya 3.3 environment. You need to run the following command:
```
ansible-playbook -i inventory.ini ./utility/rebuild-trust-stores.yml
```
Add the sitedefault.yml content from step 1 to the sitedefault.yml that forms part of the Ansible playbooks in:
```
/roles/consul/files/sitedefault.yml
```

This will result in the CA certificate being added to all trust stores on all hosts. Also, if the deployment playbook is re-run later to update the license the correct content will remain in the trust stores.

Removing from the Trust Stores

To remove a CA certificate from the trust stores, perhaps if an internal customer CA has expired, you should complete the following steps:

If the Base64 PEM encoded certificate was added to the sitedefault.yml then remove it. You need to update the sitedefault.yml files in:
```
/opt/sas/viya/config/etc/consul.d/default 
```
and
```
/roles/consul/files/sitedefault.yml
```
Use the SAS Bootstrap Config tool to remove the existing key-value pair from the cacerts tree in the SAS Configuration Server. For example, to remove my internal CA certificate added above:
```
/opt/sas/viya/home/bin/sas-bootstrap-config -token-file $CONSUL_TOKEN kv delete cacerts/MyRootCA
```
Restart the SAS Configuration Server to pick-up the changes to the sitedefault.yml file. Remember only new key-value pairs are added, existing key-value pairs will not be updated.
Use the Ansible playbook to rebuild the trust stores across all the hosts in the SAS Viya 3.3 environment. You need to run the following command:
```
ansible-playbook -i inventory.ini ./utility/rebuild-trust-stores.yml
```

This will result in the CA certificate being removed from the trust stores on all the hosts. Also, if the deployment playbook is re-run later to update the license the correct content will remain in the trust stores.

Replacing in the Trust Stores

To replace a CA certificate in the trust stores, perhaps if an internal customer CA has been renewed, you should complete the following steps:

If the Base64 PEM encoded certificate was added to the sitedefault.yml then update it. You need to update the sitedefault.yml files in:
```
/opt/sas/viya/config/etc/consul.d/default 
```
and
```
/roles/consul/files/sitedefault.yml
```
Use the SAS Bootstrap Config tool to remove the existing key-value pair from the cacerts tree in the SAS Configuration Server. For example, to remove my internal CA certificate added above:
```
/opt/sas/viya/home/bin/sas-bootstrap-config -token-file $CONSUL_TOKEN kv delete cacerts/MyRootCA
```
Restart the SAS Configuration Server to pick-up the changes to the sitedefault.yml file. Remember only new key-value pairs are added, existing key-value pairs will not be updated. Hence in step 2 you removed the existing key-value pair.
Use the Ansible playbook to rebuild the trust stores across all the hosts in the SAS Viya 3.3 environment. You need to run the following command:
```
ansible-playbook -i inventory.ini ./utility/rebuild-trust-stores.yml
```

This will result in the CA certificate being updated from the trust stores on all the hosts. Also, if the deployment playbook is re-run later to update the license the correct content will remain in the trust stores.

Programming Only

So far, everything we have discussed applies to a full deployment of SAS Viya 3.3. It is possible to deploy only the programming interfaces. This means there is no SAS Configuration Server. In such "programming only" deployments the mechanism to manage the trust stores is different. The trust stores in this case are manually updated.

The trustedcerts.pem can be edited with a text editor and CA certificates added, removed or replaced. The trustedcerts.jks must be edited using the Java Keytool utility. For example, to add an entry use the following command:

keytool -import -trustcacerts -alias <NewCA> -file <NewCA.crt> -keystore /opt/sas/viya/config/etc/SASSecurityCertificateFramework/cacerts/trustedcerts.jks

Where <NewCA.crt> is the CA certificate you want to add and <NewCA> is the alias to use for this new entry.

To remove an entry, use the following command:

keytool -delete -alias <NewCA> -keystore /opt/sas/viya/config/etc/SASSecurityCertificateFramework/cacerts/trustedcerts.jks

Where you need to know the alias of the entry to delete and this is given by <NewCA> in the example above.

To replace an entry just remove and then add it.

Any CA certificates you want to add should also be placed in the following directory:

/opt/sas/viya/config/etc/SASSecurityCertificateFramework/cacerts/

This will ensure if you re-run the Ansible deployment playbook the CA certificates will be re-added to the trust stores.

These steps will need to be repeated on all the hosts in the SAS Viya 3.3 "programming only" environment.

Conclusion

The Ansible playbooks make it easy to manage the trust stores across all the hosts in a full deployment of SAS Viya 3.3. Managing the trust stores in a "programming only" deployment of SAS Viya 3.3 is much more manual and error prone.