Welcome back! This first post serves as a follow-up to our previous post, where we initiated discussions on DevOps processes with SAS Viya CLI container images, available here, Introducing New SAS Viya CLI container image in Azure DevOps (March 2024 Update). In this two-part blog, we delve deeper into security customizations, addressing aspects such as managing refresh tokens, handling certificates not chained to public Certificate Authorities (CAs), and enhancing security measures within Azure Pipelines and Kubernetes clusters. These enhancements build upon our previous discussions, aiming to fortify your DevOps pipelines against security vulnerabilities while ensuring seamless execution. Let's explore these advancements in detail.
Here's an illustrative diagram presenting the functional flow involved.
Ensuring that only authorized entities can access these secrets is crucial for maintaining the integrity of your applications and data. One effective way to enhance security is by enforcing IP restrictions, thereby limiting access to Azure Key Vault only from specified IP addresses. In this blog example, we'll explore how to integrate IP restrictions into your Azure Pipelines to bolster the security of your Azure Key Vault.
The first step is to define a parameter in your Azure Pipeline configuration to determine whether IP restrictions should be enforced. This parameter will provide flexibility in enabling or disabling IP restrictions as per your organization's security policies. Below is an example of how to define this parameter:
- name: KV_IP_RESTRICTIONS
displayName: Do you have IP restrictions for your Azure Key Vault?
type: boolean
default: true
Before retrieving credentials from Azure Key Vault, it's essential to add the IP address of the Microsoft Agent to the allowed IP list for the Key Vault. This ensures that the pipeline can access the Key Vault securely. To achieve this, include the following Azure CLI task in your pipeline:
- task: AzureCLI@2
displayName: Allow MSFT Agent IP to Azure Key Vault
condition: ${{ eq(parameters.KV_IP_RESTRICTIONS, true) }}
inputs:
azureSubscription: '$(KV_SERVICE_CONNECTION)'
scriptType: 'bash'
scriptLocation: 'inlineScript'
inlineScript: |
# Retrieve your IP address
CURRENT_IP=$(curl -s https://ipinfo.io/ip)
az keyvault network-rule add --name ${{ parameters.AZURE_KEYVAULT }} --ip-address $CURRENT_IP > /dev/null
After retrieving credentials from Azure Key Vault, it's crucial to remove the IP address of the Microsoft Agent from the allowed IP list to maintain security posture. This step ensures that unauthorized access attempts from the Microsoft Agent IP are thwarted. Add the following Azure CLI task to your pipeline:
- task: AzureCLI@2
displayName: DisAllow MSFT Agent IP to Azure Key Vault
condition: ${{ eq(parameters.KV_IP_RESTRICTIONS, true) }}
inputs:
azureSubscription: '$(azureSubscription)'
scriptType: 'bash'
scriptLocation: 'inlineScript'
inlineScript: |
# Retrieve your IP address
CURRENT_IP=$(curl -s https://ipinfo.io/ip)
az keyvault network-rule remove --name ${{ parameters.AZURE_KEYVAULT }} --ip-address $CURRENT_IP > /dev/null
These steps ensure that only designated IP addresses, including the Microsoft Agent IP, are permitted to access sensitive information stored in Azure Key Vault. Implementing such security measures aligns with best practices for safeguarding data in the cloud and helps protect your applications from potential security threats.
In this blog example, we'll delve into how to handle IP restrictions for AKS APIs, particularly when utilizing the sas-viya-cli in a pipeline executed on a Microsoft-hosted agent.
When the AKS API is restricted based on IP addresses, it poses a challenge for pipeline execution using public IPs. We need to ensure that the Microsoft-hosted agent's IP is authorized to access the AKS API. This involves installing the Azure CLI, modifying AKS API IP restrictions, and ensuring proper permissions within the Docker image.
To accommodate these requirements, enhancements to the Docker image are necessary. Below is a snippet showcasing the modifications:
RUN dnf install -y --nodocs nodejs sudo && dnf clean all
NOTE: sudo command is mandatory to be able to install the az cli (that will be done while executing the DevOps pipeline).
This parameter will provide flexibility in enabling or disabling IP restrictions management as per your organization's security policies. Below is an example of how to define this parameter:
- name: AKS_APIS_IP_RESTRICTIONS
displayName: Do you have IP restrictions for your AKS APIs?
type: boolean
default: false
Installing the Azure CLI within the pipeline environment is mandatory to accommodate the current requirement. Here's how to do it:
- task: Bash@3
displayName: Install Azure CLI for ubi8 (RHEL8)
condition: enabled: ${{ eq(parameters.AKS_APIS_IP_RESTRICTIONS, true) }}
inputs:
targetType: 'inline'
script: |
sudo rpm --import https://packages.microsoft.com/keys/microsoft.asc
sudo dnf install -y https://packages.microsoft.com/config/rhel/8/packages-microsoft-prod.rpm
sudo dnf install -y azure-cli
To add the Microsoft agent's IP to the authorized list for AKS API access, follow these steps:
- task: AzureCLI@2
displayName: Add Agent IP to the AKS API authorized IPs
condition: ${{ eq(parameters.IP_RESTRICTIONS, true) }}
inputs:
azureSubscription: 'Service for PSGEL262'
scriptType: 'bash'
scriptLocation: 'inlineScript'
inlineScript: |
# set -x
AZURE_RG=${RACE_PREFIX}-rg
AZURE_AKS=${RACE_PREFIX}-aks
AZURE_RANGE=$(az aks show \
--resource-group $AZURE_RG \
--name $AZURE_AKS \
--query apiServerAccessProfile.authorizedIpRanges | jq -r '. | join(",")')
# Store the value to reset it later
echo "##vso[task.setvariable variable=AZURE_RANGE]$AZURE_RANGE"
# Retrieve your IP address
CURRENT_IP=$(curl -s api.ipify.org)
# Add to AKS approved list
az aks update \
--resource-group $AZURE_RG \
--name $AZURE_AKS \
--api-server-authorized-ip-ranges ${CURRENT_IP}/32,${AZURE_RANGE} --output none
sleep 10
After pipeline execution, it's essential to reset the AKS API IP restrictions to their original values to maintain the integrity of your environment:
- task: AzureCLI@2
displayName: Remove Agent IP to the AKS API authorized IPs
enabled: ${{ eq(parameters.AKS_APIS_IP_RESTRICTIONS, true) }}
inputs:
azureSubscription: '$(RG_SERVICE_CONNECTION)'
scriptType: 'bash'
scriptLocation: 'inlineScript'
inlineScript: |
# Add to AKS approved list
az aks update \
--resource-group $AZURE_RG \
--name $AZURE_AKS \
--api-server-authorized-ip-ranges $(AZURE_RANGE) --output none
By following these steps, you ensure that your pipeline can seamlessly interact with your AKS cluster while maintaining the necessary security measures.
One common method of enforcing security is by implementing IP restrictions for Kubernetes Ingress. In this blog example, we'll explore how to manage IP restrictions for Kubernetes Ingress effectively.
- name: INGRESS_IP_RESTRICTIONS
displayName: Do you have IP restrictions for your K8s Ingress?
type: boolean
default: false
Before proceeding, ensure that the kubectl command-line tool is installed while executing your pipeline. Here's how to do it:
- task: KubectlInstaller@0
displayName: Install Kubectl client
condition: ${{ or(eq(parameters.IP_RESTRICTIONS, true), eq(parameters.PUB_SIGN_CERT, false)) }}
inputs:
kubectlVersion: 'latest'
To allow the Microsoft agent's IP access to the Kubernetes Ingress, follow these steps:
- task: AzureCLI@2
displayName: Add Agent IP to the AKS Ingress authorized IPs
enabled: ${{ eq(parameters.INGRESS_IP_RESTRICTIONS, true) }}
inputs:
azureSubscription: '$(RG_SERVICE_CONNECTION)'
scriptType: 'bash'
scriptLocation: 'inlineScript'
inlineScript: |
az aks get-credentials --resource-group $AZURE_RG \
--name $AZURE_AKS \
--overwrite-existing
# kubectl config view
# Check if the property exists
load_balancer_ranges=$(kubectl get services ingress-nginx-controller -n ingress-nginx -o=jsonpath='{.spec.loadBalancerSourceRanges}' 2>/dev/null)
# Check if the output is empty or not
if [ -n "$load_balancer_ranges" ]; then
echo "Property exists, applying patch..."
kubectl patch services ingress-nginx-controller -n ingress-nginx --type='json' -p='[{"op": "add", "path": "/spec/loadBalancerSourceRanges/-", "value": "'${CURRENT_IP}/32'" }]'
else
echo "Property does not exist, creating it..."
kubectl patch services ingress-nginx-controller -n ingress-nginx --type=json '-p=[{"op": "add", "path": "/spec/loadBalancerSourceRanges", "value": ["'${CURRENT_IP}/32'"] }]'
fi
After pipeline execution, it's essential to remove the Microsoft agent's IP from the Kubernetes Ingress allowed list:
- task: Bash@3
displayName: Remove Agent IP from the AKS Ingress authorized IPs
enabled: ${{ eq(parameters.INGRESS_IP_RESTRICTIONS, true) }}
inputs:
targetType: 'inline'
script: |
# Find the index of the IP address within the loadBalancerSourceRanges
INDEX=$(kubectl get services ingress-nginx-controller -n ingress-nginx -o json | jq '.spec.loadBalancerSourceRanges | map(. == "'${CURRENT_IP}/32'") | index(true)')
# Patch the Ingress service to remove the IP address
kubectl patch services ingress-nginx-controller -n ingress-nginx \
--type=json -p='[{"op": "remove", "path": "/spec/loadBalancerSourceRanges/'$INDEX'"}]'
# Ensure the configuration is updated
kubectl patch services ingress-nginx-controller -n ingress-nginx \
--type=json -p='[{"op": "remove", "path": "/spec/loadBalancerSourceRanges/'$INDEX'"}]'
By following the steps outlined in this blog example, you can effectively control access to your applications while ensuring the necessary level of security.
By effectively managing network IP restrictions and enhancing security measures within Azure pipelines and Kubernetes clusters, organizations can fortify their DevOps pipelines against potential security threats. These practices not only ensure security but also ensure the reliability and integrity of your applications and data in cloud environments.
David Estreich (myself) related posts with Azure DevOps:
Bogdan Teleuca's other posts about SAS Viya with Azure DevOps:
The rapid growth of AI technologies is driving an AI skills gap and demand for AI talent. Ready to grow your AI literacy? SAS offers free ways to get started for beginners, business leaders, and analytics professionals of all skill levels. Your future self will thank you.