SAS Viya 2023.08 Stable release introduced an alternative to configuring System Services Security Daemon (SSSD) with Kerberos authentication to Hadoop. Kerberos authentication to Hadoop depends on Java to make the Kerberos authentication connection. The Java libraries validate the ownership of the Kerberos ticket cache, which means that the operating system of the pod needs to recognize the UID of the end-user attempting the connection. Prior to the SAS Viya 2023.08 release, this meant you had to configure SSSD for the SAS Servers and SAS Cloud Analytic Services pods. Now with SAS Viya 2023.08 and later it is possible to instead use a Name Server Switch (NSS) Wrapper to provide the user information. In this post we will examine how this can be configured and how it operates.
Concerns with SSSD
Leveraging System Services Security Daemon (SSSD) to provide user information lookup to SAS Servers and SAS Cloud Analytic Services requires running SSSD in a privilege elevated container. Running in an elevated container is against a number of recommendations for securing Kubernetes environments, such as the Cybersecurity and Infrastructure Security Agency’s Kubernetes Hardening Guidance. In addition, providing a SSSD configuration or allowing the SAS pods to generate the SSSD configuration further distributes credentials which are valid outside the Kubernetes environment. The SSSD configuration containing credentials to bind to the LDAP provider will end-up in a Kubernetes secret.
These items make configuring SSSD less than ideal. Instead, using a NSS Wrapper does not require a privilege elevated container, nor does it require any external credentials.
Configuring NSS Wrapper
The configuration of NSS Wrapper is very straightforward since you are essentially only turning on the feature. As described in the documentation, you just need to add a reference to the nss_wrapper in the transformers block of the kustomization.yaml. The reference to the nss_wrapper must come before the sas-bases/overlays/required/transformers.yaml reference. It would look like the following:
transformers:
...
- sas-bases/overlays/kerberos/nss_wrapper/add-nss-wrapper-transformer.yaml
- sas-bases/overlays/required/transformers.yaml
You then need to rebuild your site.yaml and reapply the updated site.yaml.
How NSS Wrapper Operates
The NSS Wrapper operates by using the internal SAS Logon Manager token to provide the userid. It then creates a temporary passwd type entry for the userid, uid, and gid with the following format:
<userid>:x:<uid>:<gid>:<userid>:/:/bin/false
Remember: the uid and gid values are already fetched from the Identities microservice when launching the process.
It sets two environment variables: NSS_WRAPPER_PASSWD pointing to the temporary file, and NSS_WRAPPER_GROUP pointing to /etc/group. It then leverages an environment variable LD_PRELOAD, which points to the libnss_wrapper.so shared library. More details on the libnss_wrapper.so shared library can be found here: https://cwrap.org/nss_wrapper.html. With nss_wrapper it is possible to define your own passwd and group files to be used, which means that NSS will correctly identify the user when called by the Java Hadoop client. This means the SAS containers do not need any extra privileges, to directly interact with the /etc/passwd file inside the container. Also, there is no need to configure SSSD to sit behind NSS and run SSSD in a privilege escalated container.
Validating NSS Wrapper
The functioning of the nss_wrapper is driven by the presence of an environment variable, hence why configuring the nss_wrapper is so easy. The environment variable is injected into the following containers when we add the reference to the nss_wrapper in the transformers block of the kustomization.yaml:
- SAS Compute
- SAS Launcher
- SAS/Connect
- SAS Batch
- SAS Cloud Analytic Services
This means the first and easiest way to check the nss_wrapper has been configured is to look for the environment variable. With access to kubectl we can use the following command to check for the environment variable in the SAS Cloud Analytic Services controller pod:
kubectl -n ${NS} exec -it sas-cas-server-default-controller -c sas-cas-server -- env |grep SAS_POD_USES_NSS_WRAPPER
You should see something like the following:
SAS_POD_USES_NSS_WRAPPER=true
This shows that the environment variable has been set.
Then if you want to check the processing of the nss_wrapper we can complete the following, once we have launched a SAS Compute Server pod, perhaps by logging into SAS Studio.
-
Use the following command to look at the details of the SAS Compute Server pod that has been launched by SAS Studio:
myPod=`kubectl -n ${NS} get pods -l launcher.sas.com/username=gatedemo001 -o name|cut -c 5-`; \
kubectl -n ${NS} describe pod ${myPod}|grep -E 'owner|username'
You should see something like the following:
sas.com/owner: 463452355
launcher.sas.com/username=gatedemo001
sas.com/ownerGroup: 463452355
- Use the following commands to validate the processing of the NSS Wrapper:
kubectl -n ${NS} exec -it ${myPod} -c sas-programming-environment -- id; \
kubectl -n ${NS} exec -it ${myPod} -c sas-programming-environment -- /bin/bash -c 'cat /tmp/tmp*'; \
kubectl -n ${NS} exec -it ${myPod} -c sas-programming-environment -- /bin/bash -c 'export NSS_WRAPPER_PASSWD=$(find /tmp -name tmp.*) &&
export NSS_WRAPPER_GROUP="/etc/group" &&
export LD_PRELOAD=/usr/lib64/libnss_wrapper.so &&
id'
You should see something like the following:
uid=463452355 gid=463452355 groups=463452355,1870626142
gatedemo001:x:463452355:463452355:gatedemo001:/:/bin/false
uid=463452355(gatedemo001) gid=463452355 groups=463452355,1870626142
The first id command shows that the operating system inside the pod does not recognize the user. The second command reads the content of the temporary file created for the NSS Wrapper using information from the users internal SAS Logon Manager token and information from the SAS Identities microservice. The third command then leverages the NSS Wrapper before running the id command, this illustrates that now the user is recognized.
Conclusion
With this change in the SAS Viya Stable 2023.08 release, it is now easier and more secure to complete the additional configuration required for using Kerberos to authenticate to Hadoop. The configuration of SAS Viya no-longer requires additional SSSD configuration and running privilege escalated pods.
If you want to explore this topic in more detail, you can refer to Course: Advanced Topics in Authentication on SAS Viya.
Find more articles from SAS Global Enablement and Learning here.
