How to deal with group membership limit with NFS in SAS Viya

3 Likes

When working with NFS (Network File System), it's important to be aware of certain protocol limitations that can impact how permissions are applied—especially in complex enterprise environments. One such limitation is the maximum number of auxiliary GIDs (secondary groups) that can be included in a single NFS request. According to the RFC5531 specification for AUTH_SYS/AUTH_UNIX, the limit is 16 auxiliary groups. This is a well-known constraint of the NFS protocol itself, not something specific to any vendor or implementation.

Recently, I was contacted by two different customers who ran into this exact issue. One case stood out in particular: the customer was using an Azure File Share backed by a Premium Storage Account, and sharing it over NFS for use in SAS Viya. Everything seemed to be configured correctly, but they noticed that user permissions weren’t being applied as expected. After some investigation, we discovered that the affected users belonged to more than 16 groups, and the extra group memberships were simply being ignored in the NFS request. As a result, access controls based on those additional groups failed silently.

This blog post will walk through the issue in more detail, demonstrate how it manifests in a real-world setup, and share our recommended workaround to ensure permissions are applied correctly—even when users belong to many groups. Stay tuned for the technical deep dive and practical advice!

Azure File Share

When setting up an NFS Azure File Share, one of the first and most critical decisions you'll make is selecting the appropriate Azure Storage Account type. Not all storage types support NFS, so this choice directly determines whether NFS functionality is even available. While I won’t dive into the full step-by-step process of creating an NFS-enabled Azure File Share or configuring the necessary Private Endpoint, it’s essential to understand that the storage tier matters. For example, Premium File Shares are typically required to enable NFS support. Before you begin provisioning resources, take a moment to review the storage configuration and network connectivity options.

Once everything is set up correctly, you can easily mount the NFS share using the commands provided directly in the Azure Portal.

sudo mkdir -p /mount/xxxtest/nfs-xxxfileshare
sudo mount -t aznfs xxxtest.file.core.windows.net:/xxxtest/nfs-xxxfileshare /mount/xxxtest/nfs-xxxfileshare -o vers=4,minorversion=1,sec=sys,nconnect=4

Azure NetApp Volume

To explore the 16-group limit, we had to set up an LDAP-enabled Azure NetApp Volume. In this setup, it’s crucial that the NetApp account is connected to Active Directory. This is a key requirement for creating LDAP-enabled volumes and plays an essential role in overcoming the group membership limitation. To read more about that requirement check this link.

Just like with Azure File Shares, Azure NetApp Files also provides clear mount instructions directly within the Azure Portal. When you navigate to the settings of your NetApp Volume, you'll find all the necessary details to mount the volume via NFS—making it easy to integrate with your applications or platforms like SAS Viya.

sudo mkdir test
sudo mount -t nfs -o rw,hard,rsize=262144,wsize=262144,sec=sys,vers=4.1,tcp 192.168.3.4:/test test

Jumphost as test NFS-client

I used the jump virtual machine, created with the IaC project as a test NFS-client. Important to note is that the jumphost joined as well my test AD DS domain. I did that based on the instructions I found here. Most relevant command is:

sudo realm join -v win22ad.demoad.local -U winadmin

I also edited to idmapd.conf and pointed the domain to my own created AD DS:

cat /etc/idmapd.conf

[General]
Verbosity = 0
# set your own domain here, if it differs from FQDN minus hostname
Domain = demoad.local

[Mapping]

Nobody-User = nobody
Nobody-Group = nogroup

To make sure the windows server where I installed AD DS was found I added extra line in the /etc/hosts file:

192.168.0.11 win22ad.demoad.local

After that the same users I will use in Viya, with same UIDs and GIDs can be used on this machine.

Mounting Azure File Share and Azure NetApp Volume on the Jumphost

Now that the jumphost joined the AD DS domain I mounted both volumes on that NFS-client. Before I jumped to Viya I wanted to experience the 16-group limitation with Azure File Shares.

sudo mount -t aznfs xxxtest.file.core.windows.net:/xxxtest/nfs-xxxfileshare /mount/xxxtest/nfs-xxxfileshare -o vers=4,minorversion=1,sec=sys,nconnect=4

sudo mount -t nfs -o rw,hard,rsize=262144,wsize=262144,sec=sys,vers=4.1,tcp 192.168.3.4:/test /mount/test/

ls -ltr xxxtest/nfs-xxxfileshare/frederik-test/
drwxr-x--- 2 michael group20 64 Sep 17 13:48 michael-from-linux

ls -ltr test
drwxr-x--- 2 michael group20 4096 Sep 22 17:54 michael-from-linux

To illustrate the issue, I created a subfolder named michael-from-linux on both mounted volumes. This folder is owned by the user Michael, but I expected Frederik (who belongs to group20) to have read access based on group permissions. This setup was designed to test whether NFS would correctly honor access rights when the user is part of a secondary group—especially in light of the 16-group limitation.

The extended groups issue

Next, I switched to the Frederik user and attempted to list the contents of the `michael-from-linux` folder. As it turns out, access was only granted when using the LDAP-enabled Azure NetApp Volume. On the Azure File Share, the permissions failed to apply correctly due to the 16-group limitation and was exactly the issue our customer had been facing in their Viya environment.

sudo su frederik

ls -ltr xxxtest/nfs-xxxfileshare/frederik-test/michael-from-linux/
ls: cannot open directory 'xxxtest/nfs-xxxfileshare/frederik-test/michael-from-linux/': Permission denied

ls -ltr test/michael-from-linux/
total 0
-rw-r--r-- 1 michael domain users 0 Sep 22 17:54 mikes.txt

NFS mounts in Viya

I wanted to end our test by showing how user Frederik experience the issue from in SAS Viya. Remember that the Viya identities services is connected to the same AD DS. So I can logon with same user Frederik in SAS Viya.

I added the NFS-mounts to the sas-compute-job-config podtemplate that a SAS Compute session is using when launching the sas-compute-server pod:

name: nfs-xxxfileshare
nfs:
path: /xxxtest/nfs-xxxfileshare
server: 192.168.2.6
name: nfs-netapp
nfs:
path: /test
server: 192.168.3.4

mountPath: /mnt/xxxfileshare
name: nfs-xxxfileshare
mountPath: /mnt/xxx-netapp
name: nfs-netapp

I moved over to SAS Studio to validate the behavior from within the SAS Viya environment. Using kubectl exec command, I accessed the shell of the associated sas-computer-server pod — allowing me to simulate user-level access directly from the container. No surprises here: when my user attempted to list the contents of the michael-from-linux folder on both mounts, the results were consistent with what we had seen earlier.

kubectl exec -it -n viya4 sas-compute-server-xxxxx-xxxx-xxx sas-programming-environment -- /bin/bash

bash-4.4$ df -h
Filesystem Size Used Avail Use% Mounted on
192.168.2.6:/xxxtest/nfs-xxxfileshare 100G 0 100G 0% /mnt/xxxfileshare
192.168.3.4:/test 50G 0 50G 0% /mnt/xxx-netapp

bash-4.4$ whoami
whoami: cannot find name for user ID 208001108
bash-4.4$ id
uid=208001108 gid=208000513 groups=208000513,208004101,208004601,208004602,208004603,208004604,208004606,208004607,208004608,208004609,208004610,208004611,208004612,208004613,208004614,208004615,208004616,208004617,208004618,208004619,208004620,208005101
bash-4.4$ ls -ltr /mnt/xxxfileshare/frederik-test/michael-from-linux/
ls: cannot open directory '/mnt/xxxfileshare/frederik-test/michael-from-linux/': Permission denied
bash-4.4$ ls -ltr /mnt/xxx-netapp/michael-from-linux/
total 0
-rw-r--r-- 1 nobody 208000513 0 Sep 22 17:54 mikes.txt
bash-4.4$

One interesting detail I noticed while working inside the pod shell was how UIDs and GIDs are displayed depending on the storage type and identity resolution. When the UID or GID is recognized by SAS Viya, thanks to the identities service retrieving it as a `posixAccount` or `posixGroup`, the `ls` command shows the owner or group as `nobody`. This behavior occurs specifically with the Azure NetApp Volume. In contrast, when listing the same folder on the Azure File Share, the actual numeric values for UID and GID are shown instead.

bash-4.4$ ls -ltr /mnt/xxxfileshare/frederik-test
total 1
drwx------ 2 208001108 208000513 64 Sep 17 13:46 fred-from-linux
drwxr-x--- 2 208001111 208004620 64 Sep 17 13:48 michael-from-linux
bash-4.4$ ls -ltr /mnt/xxx-netapp
total 4
drwxr-x--- 2 nobody nobody 4096 Sep 22 17:54 michael-from-linux

And in SAS Studio, I tested the same scenario from the perspective of the Frederik user. The results were consistent: when trying to open the michael-from-linux folder on the Azure File Share, Frederik encountered permission issues—despite being a member of the correct group. However, when accessing the same folder on the Azure NetApp Volume, everything worked as expected. The permissions were correctly applied, and Frederik was able to open the folder without any issues.

Conclusion

I hope this deep dive has helped clarify what’s meant by the extended group issue that can occur on NFS clients—especially when users belong to more than 16 groups. Through our testing and real-world scenarios, it became clear that this is not just a theoretical limitation, but one that can directly impact access control in production environments.

The key takeaway? LDAP integration with Azure NetApp Volumes provides a reliable and effective workaround. By leveraging Active Directory and LDAP, you can bypass the NFS protocol’s 16-group limit and ensure that group-based permissions are applied correctly, even in complex identity setups.

If you're running SAS Viya or any other workload that relies heavily on NFS and granular access control, this is a configuration worth considering.