We are running SAS Viya 4 LTS 2024.09 on Azure Kubernetes Service (AKS) with an Azure Managed Lustre File System (AMLFS) mounted to the cluster using Container Storage Interface (CSI) drivers. I am posting this to warn anyone who might be using CSI drivers that a recent update to the driver may cause Kubernetes to systematically modify group ownerships on files and directories to GID 1001 as well as possibly change permissions when SAS Viya pods attempt to mount the Lustre file system.
Previous to this incident, we were running version 0.1.15 of the Azure Lustre CSI driver. This was updated to 0.1.18. Prior to version 0.1.17, the “fsGroupChangePolicy: OnRootMismatch” spec on the SAS pods prevented Kubernetes from changing group ownerships/permissions on the AMLFS. Starting with that version, however, this spec is ignored in favor of the "fsGroupPolicy" spec on the driver itself. (filename is csi-azurelustre-driver.yaml). In order to stop Kubernetes from making these changes, the fsGroupPolicy spec must be set to None. Without that line, the default behavior is File, which "indicates that the CSI volume driver supports volume ownership and permission change via fsGroup, and Kubernetes may use fsGroup to change permissions and ownership of the volume to match user requested fsGroup in the pod's SecurityPolicy regardless of fstype or access mode." Updateing the driver does not add the fsGroupPolicy - this must be done manually or these changes will start being made as soon as pods start trying to mount the file system. Here is the updated csi-azurelustre-driver.yaml file:
---
apiVersion: storage.k8s.io/v1
kind: CSIDriver
metadata:
name: azurelustre.csi.azure.com
spec:
attachRequired: false
podInfoOnMount: true
fsGroupPolicy: None
This change was made in response to this issue: Containers running in AKS can't write to lustre if the user is not root. · Issue #194 · kubernetes-s...
Also see the Kubernetes CSI Developer Documentation here: FSGroup Support - Kubernetes CSI Developer Documentation
