SAS Viya on Azure: Firewall rules for NGINX when leveraging a SCIM client
SAS Viya on Azure Kubernetes Service uses the NGINX Ingress Controller. Several times I have seen confusion over the firewall settings that prevent access to the Ingress Controller. This often causes issues when trying to configure the SAS Viya environment for SCIM, where the SCIM client needs to be able to directly access the Identities microservice. In this blog I want to explore the settings for the Ingress Controller that impact the firewall settings and show where these are defined. From this we will see where the confusion can arise and how to resolve these issues.
The Setup
To explore this topic, we will take a deployment of SAS Viya LTS 2022.09 in AKS. This environment has used the SAS Viya 4 Infrastructure as Code (IaC) for Microsoft Azure to provision the AKS cluster. The Ingress Controller is then installed into the cluster using HELM. In this case the following command is used to install the Ingress Controller:
helm install ingress-nginx --namespace ingress-nginx \
--set controller.service.externalTrafficPolicy=Local \
--set controller.service.sessionAffinity=None \
--set controller.config.use-forwarded-headers="true" \
--set controller.autoscaling.enabled=true \
--set controller.autoscaling.minReplicas=2 \
--set controller.autoscaling.maxReplicas=5 \
--set controller.resources.requests.cpu=100m \
--set controller.resources.requests.memory=500Mi \
--set controller.autoscaling.targetCPUUtilizationPercentage=90 \
--set controller.autoscaling.targetMemoryUtilizationPercentage=90 \
--set-string controller.service.annotations."service\.beta\.kubernetes\.io/azure-allowed-service-tags"="AzureActiveDirectory" \
--set controller.service.loadBalancerSourceRanges="254.254.254.254/32"
--version 4.4.2 \
ingress-nginx/ingress-nginx
This HELM installation command should allow access to the Ingress Controller from both the IP address 254.254.254.254 and everything covered by the Azure Service Tag of "AzureActiveDirectory". We have used just the IP address 254.254.254.254 in the LoadBalancer Source Ranges as it will make it easier to see where this value is used. In most SAS Viya environments this will instead be a list of different IP address ranges where your various end-users will be connecting from.
The Allowed Service Tag setting is useful in Microsoft Azure since it allows us to define firewall rules dependent on the tags maintained by Microsoft. Using the tags means that we do not need to constantly update the IP address ranges used by those services when they are changed by Microsoft.
We can then review the deployed Ingress Controller service within AKS with the following command:
kubectl -n ingress-nginx describe svc ingress-nginx-controller
Which will show:
Name: ingress-nginx-controller
Namespace: ingress-nginx
Labels: app.kubernetes.io/component=controller
app.kubernetes.io/instance=ingress-nginx
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=ingress-nginx
app.kubernetes.io/part-of=ingress-nginx
app.kubernetes.io/version=1.5.1
helm.sh/chart=ingress-nginx-4.4.2
Annotations: meta.helm.sh/release-name: ingress-nginx
meta.helm.sh/release-namespace: ingress-nginx
service.beta.kubernetes.io/azure-allowed-service-tags: AzureActiveDirectory
Selector: app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
Type: LoadBalancer
IP Family Policy: SingleStack
IP Families: IPv4
IP: 10.0.63.178
IPs: 10.0.63.178
LoadBalancer Ingress: 20.232.249.245
Port: http 80/TCP
TargetPort: http/TCP
NodePort: http 30207/TCP
Endpoints: 10.244.1.2:80,10.244.6.47:80
Port: https 443/TCP
TargetPort: https/TCP
NodePort: https 31352/TCP
Endpoints: 10.244.1.2:443,10.244.6.47:443
Session Affinity: None
External Traffic Policy: Local
HealthCheck NodePort: 31529
LoadBalancer Source Ranges: 254.254.254.254/32
This includes the information we expect with the LoadBalancer Source Ranges and the annotation for the Allowed Service Tags.
If we look at the corresponding Network Security Group within the Azure Portal (https://portal.azure.com) we will see that the following firewall rules have been automatically defined for us:
Select any image to see a larger version.
Mobile users: To view the images, select the "Full" version at the bottom of the page.
Again, this reflects the information we provided when installing with HELM and we would expect to be allowed access from both IP address 254.254.254.254 and everything covered by the Azure Service Tag of "AzureActiveDirectory".
The Problem
In our example setup we would expect that Azure Active Directory is able to access the Ingress Controller. This is a core requirement for configuring SCIM with SAS Viya. However, when we try and provide the Tenant URL, Secret Token, and test the connection we will instead see an error:
This rather vague error just tells us there is a problem connecting from Azure Active Directory to our Identities microservice. This could be a network problem or a problem with the Secret Token. But if we have already validated the Secret Token using CURL when we generated it, we can eliminate this as the cause of the problem. This means the problem must lie with the network connection from Azure Active Directory to SAS Viya.
The Cause
Everything that we have shown for our setup would suggest that Azure Active Directory should have no problems connecting. But something is preventing the connection. When we use the LoadBalancer Source Ranges, in the background in addition to defining the Network Security Group rules, additional IPTABLES firewall rules are also defined on the AKS nodes where the Ingress Controller service is running.
We can demonstrate this by using the kubectl-exec project to run a debugging privileged container on the AKS node and access the IPTABLES rules on that node. Since this is a privileged container, you should not run this on a production cluster. We are running it here on our testing cluster to demonstrate what is happening. Once you have installed kubectl-exec as per the instructions you will need to edit the /usr/local/bin/kubectl-exec script to allow the NET_ADMIN capability, as discussed here. The changed part of the file will look like the following:
...
#nsenter JSON overrrides
OVERRIDES="$(cat <<EOT
{
"spec": {
"nodeName": "$NODE",
"hostPID": true,
"containers": [
{
"securityContext": {
"privileged": true,
"capabilities": {
"add": [ "NET_ADMIN" ]
}
},
...
Then you can use the following command to find the node where your Ingress Controller is running:
kubectl -n ingress-nginx get pods -o wide
Which will output something like the following:
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
ingress-nginx-controller-7f5d995f47-d9858 1/1 Running 0 9h 10.244.6.47 aks-generic-21185651-vmss000000
ingress-nginx-controller-7f5d995f47-wml2w 1/1 Running 0 10h 10.244.1.2 aks-generic-21185651-vmss000002 The kubectl-exec script can then be used to launch the debugging container on the node with the following command:
kubectl-exec aks-generic-21185651-vmss000000
Finally, you can use the following IPTABLES command to list all the rules in the NAT table and search for the IP address we included in the LoadBalancer Source Ranges:
iptables -t nat -L|grep -A1 -B1 254.254.254.254
Which will output something like the following:
target prot opt source destination
KUBE-EXT-CG5I4G2RS3ZVWGLK all -- 254.254.254.254 anywhere /* ingress-nginx/ingress-nginx-controller:http loadbalancer IP */
KUBE-MARK-DROP all -- anywhere anywhere /* ingress-nginx/ingress-nginx-controller:http loadbalancer IP */
--
target prot opt source destination
KUBE-EXT-EDNDUDH2C75GIR6O all -- 254.254.254.254 anywhere /* ingress-nginx/ingress-nginx-controller:https loadbalancer IP */
KUBE-MARK-DROP all -- anywhere anywhere /* ingress-nginx/ingress-nginx-controller:https loadbalancer IP */
This shows that we have two rules defined for the IP address that allow access via HTTP and HTTPS coming from 254.254.254.254/32, but all other source ranges will be dropped by IPTABLES. Therefore, this is the cause of our network connection issue. The connection from Azure Active Directory is allowed by the Network Security Group rules but is being dropped by IPTABLES on the actual node where the Ingress Controller is running.
The Solution
To resolve this issue, we need to remove the LoadBalancer Source Ranges from the Ingress Controller service definition. If we have allowed access to the AKS control plane from our own machine we could perform this in a browser using the Azure Portal (https://portal.azure.com), or we could use kubectl from our management host, or we could use HELM to update the deployment of the Ingress Controller. For example, to use kubectl to edit the service definition use the following command:
kubectl -n ingress-nginx edit svc ingress-nginx-controller
This will open the service definition for editing and we can just remove the required lines. As soon as we save the edited content it is applied and the IPTABLES rules will be removed alongside the Network Security Group rules.
CAUTION 1: If you only had the LoadBalancer Source Ranges defined and you have now removed them, Microsoft Azure will helpfully define a new Network Security Group rule allowing access to 0.0.0.0 which is everyone on the internet.
CAUTION 2: If you have both the LoadBalancer Source Ranges defined and the Allowed Service Tags, when you remove the LoadBalancer Source Ranges, Microsoft Azure will only have a Network Security Group rule allowing access from the service tags. This means that your end-users will not be able to access your SAS Viya environment. You will need to manually add Network Security Group rules to allow access to your environment. You can add these rules either through the Azure Portal (https://portal.azure.com) or using the Azure CLI.
Conclusion
As we have seen leveraging the LoadBalancer Source Ranges makes it easy to secure your Ingress Controller. However, this can have unintended consequences when you later need to open access to the Ingress Controller to other services running in Microsoft Azure. As such, if you will configure your SAS Viya environment running in Azure for SCIM with Azure Active Directory, you are better using the Allowed Service Tags to allow Azure Active Directory access to your Ingress Controller. Then manually, defining the Network Security Group rules to allow your end-users access.
Find more articles from SAS Global Enablement and Learning here.
