SAS Viya 2020.1.4 (and later) Kerberos Delegation with Hadoop

Continuing the series of posts about Kerberos delegation, this time we will examine Kerberos delegation and accessing a Secured Hadoop cluster. So far in this series we have looked at an overview of Kerberos delegation, the process flow for unconstrained delegation, and the process flow for constrained delegation, and the standard configuration settings for Kerberos delegation. In this article we will cover some important considerations for Kerberos delegation to Hadoop and look at the additional configuration required.

Considerations for Kerberos Delegation to Hadoop

First and foremost, only unconstrained Kerberos delegation is supported when accessing a secured Hadoop cluster. The Hadoop client is a Java application and Java does not currently completely support the use of constrained delegation credentials. This can result in connections to the Hadoop environment occurring as the service rather than the individual. Requests have been made to the Java developer community to resolve the issues in Java, but at the time of writing these have yet to be resolved.

Next the Hadoop Java client makes operating system calls to identify the user of the process. This means that the end-user running the process must be recognized by the operating system. So, both the username and numerical uid must resolve with the Name Server Switch (NSS) within the pod where the process is running. This impacts SAS Compute Server, SAS Batch Jobs, and SAS Cloud Analytic Services. The pods of these services must have NSS configured to identify the end-users. As such, additional configuration is required to enable System Services Security Daemon (SSSD) in these pods which will provide the user information to NSS.

The configuration of SSSD requires an Active Directory or LDAP Server to hold this user information. Also, with the addition of Azure Active Directory Domain Services, the same can be done with Azure Active Directory. Ideally this LDAP provider will also be the LDAP provider used by the Viya Identities microservice and will provide uid/gid information to both SSSD and the Identities microservice.

However, if the LDAP provider does not hold uid/gid information then both SSSD and the Identities microservice can generate uid/gid information. This can lead to a problem where the uid generated by the Identities microservice does not match the uid generated by SSSD. For the case of Active Directory, SSSD will use the Windows Security Identifier (SID) to generate the uid, while the Identities microservice will use a SHA-256 bit hash of the sAMAccountName. As such, the override settings for the Identities microservice can be used to provide the SSSD generated uid for the end-user. In a future release of Viya, the Identities microservice will have a REST API for loading identifier information, and this will be incorporated into the SAS Viya CLI.

Additional Configuration

Now that we understand the considerations for Kerberos delegation to Hadoop, we can examine the additional configuration required on top of the standard unconstrained delegation configuration. We will not address the basic configuration of SAS/ACCESS to Hadoop, which is covered in the documentation. The additions that must be configured for unconstrained delegation to Hadoop are:

SSSD configuration for SAS Compute Server and SAS Batch Server
SSSD configuration for SAS Cloud Analytic Services

We will address SAS/CONNECT with Kerberos in a separate post. The README.md in sas-bases/examples/kerberos/sas-servers provides the following information:

**Note:** If SAS/ACCESS Interface to Hadoop will be used with a Hadoop deployment that is Kerberos-protected, System Security Services Daemon (SSSD) is required to be configured. To configure SSSD for SAS Compute Server and SAS Batch Server, follow the instructions in `$deploy/sas-bases/examples/kerberos/sssd/README.md`. For CAS, follow the instructions in `$deploy/sas-bases/overlays/cas-server/README.md`. For SAS/CONNECT, follow the instructions in `$deploy/sas-bases/examples/sas-connect-spawner/README.md`.

We will start with SAS Compute Server and SAS Batch Server. The README.md in sas-bases/examples/kerberos/sssd informs us that we just need to add `sas-bases/overlays/kerberos/sssd/add-sssd-container-transformer.yaml` to the transformers block of the base kustomization.yaml file. However, and the README.md is quite clear, the positioning of the additional line inside the transformers block is very important. To quote the README.md:

**Important:** This line must come before any network transformers (transformers that start with "- sas-bases/overlays/network/") and the required transformer ("- sas-bases/overlays/required/transformers.yaml"). Note that your configuration may not have network transformers if security is not configured. This line must also be placed after any Kerberos transformers (transformers starting with "- sas-bases/overlays/kerberos/sas-servers").

So, we will have:

        transformers:
        ...
        # Place after any sas-bases/overlays/kerberos lines
        - sas-bases/overlays/kerberos/sssd/add-sssd-container-transformer.yaml
        # Place before any sas-bases/overlays/network lines and before
        # sas-bases/overlays/required/transformers.yaml

For example, in an environment with Full-Stack TLS we would have the following at the start of the transformers block of the kustomization.yaml:

transformers:
  - sas-bases/overlays/kerberos/sas-servers/cas-kerberos-tls-transformer.yaml
  - sas-bases/overlays/kerberos/sas-servers/sas-kerberos-deployment-tls.yaml
  - sas-bases/overlays/kerberos/sas-servers/sas-kerberos-job-tls.yaml
  - sas-bases/overlays/kerberos/sssd/add-sssd-container-transformer.yaml
  - sas-bases/overlays/network/ingress/security/transformers/product-tls-transformers.yaml   # TLS
  - sas-bases/overlays/network/ingress/security/transformers/ingress-tls-transformers.yaml   # TLS
  - sas-bases/overlays/network/ingress/security/transformers/backend-tls-transformers.yaml   # TLS
  - sas-bases/overlays/required/transformers.yaml
  - sas-bases/overlays/kerberos/http/transformers.yaml

For SAS Cloud Analytic Services, the README.md in sas-bases/examples/cas/configure/ tells us that we must add `sas-bases/overlays/cas-server/cas-sssd-sidecar.yaml` to the transformers block. Again, the placement of this line inside the transformers block is important and it must appear before the entry `sas-bases/overlays/required/transformers.yaml` and any TLS transformers. Also, in testing if this appeared after the Kerberos transformers an error was generated trying to apply the site.yaml. So now our example with Full-Stack TLS will have the following at the start of the transformers block of the kustomization.yaml:

transformers:
  - sas-bases/overlays/cas-server/cas-sssd-sidecar.yaml
  - sas-bases/overlays/kerberos/sas-servers/cas-kerberos-tls-transformer.yaml
  - sas-bases/overlays/kerberos/sas-servers/sas-kerberos-deployment-tls.yaml
  - sas-bases/overlays/kerberos/sas-servers/sas-kerberos-job-tls.yaml
  - sas-bases/overlays/kerberos/sssd/add-sssd-container-transformer.yaml
  - sas-bases/overlays/network/ingress/security/transformers/product-tls-transformers.yaml   # TLS
  - sas-bases/overlays/network/ingress/security/transformers/ingress-tls-transformers.yaml   # TLS
  - sas-bases/overlays/network/ingress/security/transformers/backend-tls-transformers.yaml   # TLS
  - sas-bases/overlays/required/transformers.yaml
  - sas-bases/overlays/kerberos/http/transformers.yaml

With this default configuration the values used to generate the SSSD configuration are loaded from the Identities microservice LDAP configuration. This works well when the LDAP Server contains and provides values for "uid". However, if the LDAP provider does not have values for the end-user uid, a custom SSSD configuration will need to be used. To deploy a custom SSSD configuration file complete the following:

Create a directory under site-config to hold the customized files: site-config/kerberos/sssd/
Copy the files in the sas-bases/examples/kerberos/sssd directory to the site-config/kerberos/sssd directory
Copy your custom sssd.conf to site-config/kerberos/sssd/sssd.conf
For SAS Compute Server and SAS Batch Server add the following to the generators block of the kustomization.yaml:
```
generators:
  ...
  - site-config/kerberos/sssd/secrets.yaml
```

For SAS Cloud Analytic Services add the following to the kustomization.yaml:

secretGenerator:
  - name: sas-sssd-config
    files:
      - SSSD_CONF=site-config/kerberos/sssd/sssd.conf
    type: Opaque

For SAS Compute Server and SAS Batch Server add the following to the transformers block of the kustomization.yaml:
```
  - sas-bases/overlays/kerberos/sssd/add-sssd-configmap-transformer.yaml
```
The new line must come after the sas-bases/overlays/kerberos/sssd/add-sssd-container-transformer.yaml line
For SAS Cloud Analytic Services add the following to the transformers block of the kustomization.yaml:
```
  - site-config/cas-server/cas-sssd-example.yaml 
```

This means that the top of the transformers block of the kustomization.yaml for our environment with Full-Stack TLS will now look like:

transformers:
  - sas-bases/overlays/cas-server/cas-sssd-sidecar.yaml
  - site-config/cas-sssd-example.yaml
  - sas-bases/overlays/kerberos/sas-servers/cas-kerberos-tls-transformer.yaml
  - sas-bases/overlays/kerberos/sas-servers/sas-kerberos-deployment-tls.yaml
  - sas-bases/overlays/kerberos/sas-servers/sas-kerberos-job-tls.yaml
  - sas-bases/overlays/kerberos/sssd/add-sssd-container-transformer.yaml
  - sas-bases/overlays/kerberos/sssd/add-sssd-configmap-transformer.yaml
  - sas-bases/overlays/network/ingress/security/transformers/product-tls-transformers.yaml   # TLS
  - sas-bases/overlays/network/ingress/security/transformers/ingress-tls-transformers.yaml   # TLS
  - sas-bases/overlays/network/ingress/security/transformers/backend-tls-transformers.yaml   # TLS
  - sas-bases/overlays/required/transformers.yaml
  - sas-bases/overlays/kerberos/http/transformers.yaml

With all this in place the standard commands can be used to build the new site.yaml and apply it to the environment.

This means that a new SSSD sidecar container will be run in the pods for SAS Compute Server, SAS Batch Server, and SAS Cloud Analytic Services. This SSSD sidecar will allow the operating system in these pods to identify the end-users. For example, once SAS Cloud Analytic Services has been restarted to pick-up the changes the following command can be used to validate the user attributes for an end-user:

NS={{ViyaNamespace}};\
kubectl -n ${NS} exec -it sas-cas-server-default-controller -c cas -- id {{username}}

You should see something like the following:

uid=1094001468(gatedemo001) gid=1094002622(GEL_SAS) groups=1094002622(GEL_SAS),1094001467(GEL Viya Users),1094001845(GEL CAS Host Launch)

Which validates that SSSD is operating correctly behind NSS to provide the user attributes.

Overriding Identities Identifier

As we discussed above, if the LDAP provider does not hold the end-user uid information then both SSSD and the Identities microservice will automatically generate a uid for end-user. However, the generated uid values will not match, and this will prevent the Java Hadoop client from working. In such a case we need to provide the override information to the Identities microservice. This could be provided with the sitedefault.yaml or manually using the SAS Bootstrap CLI. For example, to use the SAS Bootstrap CLI to set the override for the gatedemo001 user listed above complete the following:

Use the following command to exec into the SAS Consul Server pod:

NS={{ViyaNamespace}};\
kubectl -n ${NS} exec -it sas-consul-server-0 -- /bin/bash

Use the following commands to set the two required environment variables:

export CONSUL_HTTP_ADDR=https://localhost:8500
export CONSUL_HTTP_TOKEN=`cat /opt/sas/viya/config/etc/SASSecurityCertificateFramework/tokens/consul/default/client.token`

Use the following command to set the override uid value for gatedemo001 based on the value observed earlier from SSSD and exit the SAS Consul Server pod:
```
/opt/sas/viya/home/bin/sas-bootstrap-config kv write config/identities/sas.identities.identifier.users.override gatedemo001=1094001468 ;\
exit
```
Use the following command to restart the Identities microservice:
```
NS={{ViyaNamespace}};\
kubectl -n ${NS} delete pod -l app=sas-identities;\
kubectl -n ${NS} wait --for=condition=Ready pod -l app=sas-identities --timeout=240s
```
You should see something like the following:
```
pod "sas-identities-7b8776587c-x4f2d" deleted
pod/sas-identities-7b8776587c-g2jj4 condition met
```
The Identities microservice will take a couple of minutes to restart, if the Identities microservice does not yet show ready re-run the following command until it does show ready:
```
kubectl -n ${NS} wait --for=condition=Ready pod -l app=sas-identities --timeout=240s
```
This completes applying the override uid value for the gatedemo001 user.

Conclusion

In this post we have shown that careful consideration needs to be given before attempting to configure Kerberos unconstrained delegation to Hadoop. First ensure that unconstrained delegation is suitable; remember Microsoft Windows Credential Guard will prevent unconstrained delegation working. Secondly ensure that the correct `uid` value is returned by both SSSD and the Identities microservice. If the `uid` values do not match the Java-based Hadoop client will fail.

Given these considerations, the actual configuration for SAS Viya is relatively straightforward. However, care must be taken in constructing the kustomization.yaml to ensure new entries are placed in the correctly order, especially within the transformers block of the kustomization.yaml.