In the fast-paced world of software development, streamlining workflows and ensuring consistency across environments are critical challenges. Azure DevOps, Microsoft's comprehensive set of development tools, provides a solution to these challenges through the use of container jobs. In this post, we'll explore why container jobs are essential, how to build and store containers, and the best practices for designing YAML files to consume them effectively.
In this post, I will illustrate the concept through the following SAS common use case:
How can we integrate SAS Viya CLI features into your Azure DevOps pipeline?
Below are a few supplementary constraints:
Containerization is a game-changer in software development, offering a lightweight and portable solution for packaging, distributing, and running applications. Azure DevOps leverages containerization through container jobs, providing a consistent environment for your builds and releases.
Select any image to see a larger version.
Mobile users: To view the images, select the "Full" version at the bottom of the page.
To use container jobs effectively, you first need to build your container. Azure DevOps supports various container technologies such as Docker. In your project repository, create a Dockerfile specifying the dependencies and configurations required for your application.
The Dockerfile is set to install the SAS Viya CLI, along with all accessible plugins and the pyviyatools project from GitHub, ensuring the installation of all necessary prerequisites for these tools. Additionally, it will address context prerequisites, enabling container job execution by configuring the azpcontainer user with all required permissions and binaries and ensuring the execution of the node command, a prerequisite for Azure DevOps.
See Non glibc-based containers for guidelines.
# Step 0: RedHat UBI8 base image
FROM registry.access.redhat.com/ubi8/ubi
USER 0
# Step 1: Install the container job prerequisites
RUN dnf install -y --nodocs bind-utils sudo git python3 nodejs tar iputils jq && dnf clean all
RUN groupadd -g 123 docker_azpcontainer \
&& useradd -m -g 123 -u 1001 vsts_azpcontainer
# Step 2: Copy and install your certificates (if you need)
# COPY SAS_SHA2_Issuing_CA02_full_chain.pem /etc/pki/ca-trust/source/anchors
# RUN update-ca-trust
# Step 3: Install the Azure CLI
RUN rpm --import https://packages.microsoft.com/keys/microsoft.asc \
&& dnf install -y https://packages.microsoft.com/config/rhel/8/packages-microsoft-prod.rpm \
&& dnf install -y azure-cli
# Step 4: Copy and install the SAS Viya CLI
COPY sas-*.tgz /tmp/
RUN mkdir -p /tmp/utilities \
&& tar -xvf /tmp/sas-viya-cli-*.tgz -C /tmp/utilities \
&& mv /tmp/utilities/sas-viya /usr/local/bin \
&& rm -Rf /tmp/utilities \
&& chmod +x /usr/local/bin/sas* \
&& rm -f /tmp/sas-* \
&& ln -s /usr/bin/python3 /usr/bin/python
# Step 5 (optional): Install pyviyatools prerequisites
RUN pip3 install requests argparse configobj
USER 1001
WORKDIR /home/vsts_azpcontainer
# Step 6: Install All plugins of the sas-viya CLI
RUN sas-viya plugins install --repo SAS all
# Step 7: Install pyviyatools from GitHub
RUN git clone https://github.com/sassoftware/pyviyatools.git
#LABEL "com.azure.dev.pipelines.agent.handler.node.path"="/usr/bin/node"
USER 0
CMD [ "node" ]
Step 0: I decided to use the ubiI8 image as this is a base image mostly used by SAS in SAS Viya.
Step 1: While the pipeline is in progress, Azure DevOps will attempt to execute the container under the user called vsts_azpcontainer. To ensure that all SAS Viya plugins are installed in the user's directory, I opted to establish the user during the build phase and supply the plugins to the user's home directory.
Build your Docker image (first locally) using the following command:
docker build -t sas-viya-cli:latest .
When designing a container, a frequently used strategy is to first build it locally on a Linux-based system with Docker installed (I am using WSL) or using Docker for Desktop. Afterward, validation tests are carried out prior to pushing the updated image version to a central repository (in our case an Azure Container Registry).
Note: The most frequent issues often come from communication problems, network issues, or SSL certificate-related issues.
Azure Container Registry (ACR) is a fully managed Docker registry service that makes it easy to manage container images. Here is a snippet to be able to build & push your image to your ACR :
az login # or az logion --use-device-code
az account set --subscription "myownsubscription"
az acr build --registry myownacr --image viya-tool/sas-viya-cli:latest .
Now, let's design the YAML file to use the container in your Azure DevOps pipeline.
variables:
azureSubscription: Service for Pipeline Execution
trigger:
- none
jobs:
- job: runViyaCliLogin
container:
image: myownacracr.azurecr.io/viya-tool/sas-viya-cli:latest
endpoint: 'ACR Connection'
steps:
- task: KubectlInstaller@0
inputs:
kubectlVersion: 'latest'
- task: AzureCLI@2 # First task referenced in the explanation bellow
inputs:
azureSubscription: '$(azureSubscription)'
scriptType: 'bash'
scriptLocation: 'inlineScript'
inlineScript: |
# set -x
AZURE_RG=sasviyarg
AZURE_AKS=sasviyaaks
AZURE_RANGE=$(az aks show \
--resource-group $AZURE_RG \
--name $AZURE_AKS \
--query apiServerAccessProfile.authorizedIpRanges \
| jq -r '. | join(",")')
# Store the value to reset it later
echo "##vso[task.setvariable variable=AZURE_RANGE]$AZURE_RANGE"
# Retrieve your IP address
CURRENT_IP=$(dig +short "myip.opendns.com" "@resolver1.opendns.com")
# Add to AKS approved list
az aks update -g $AZURE_RG -n $AZURE_AKS --api-server-authorized-ip-ranges ${CURRENT_IP}/32,${AZURE_RANGE}
az aks get-credentials --resource-group $AZURE_RG \
--name $AZURE_AKS \
--overwrite-existing
kubectl patch services ingress-nginx-controller -n ingress-nginx \
--type=json -p='[{"op": "add", "path": "/spec/loadBalancerSourceRanges/-", "value": "'${CURRENT_IP}/32'" }]'
enabled: false
displayName: Add Agent IP to the AKS Cluster (IPs)
- task: Bash@3 # Second task referenced in the explanation bellow
inputs:
targetType: 'inline'
script: |
sas-viya --help
- task: AzureCLI@2 # Third task referenced in the explanation bellow
inputs:
azureSubscription: '$(azureSubscription)'
scriptType: 'bash'
scriptLocation: 'inlineScript'
inlineScript: |
# set -x
AZURE_RG=sasviyarg
AZURE_AKS=sasviyaaks
# Retrieve your IP address
CURRENT_IP=$(dig +short "myip.opendns.com" "@resolver1.opendns.com")
INDEX=$(kubectl get services ingress-nginx-controller -n ingress-nginx -o json | jq '.spec.loadBalancerSourceRanges | map(. == "'${CURRENT_IP}/32'") | index(true)')
kubectl patch services ingress-nginx-controller -n ingress-nginx \
--type=json -p='[{"op": "remove", "path": "/spec/loadBalancerSourceRanges/'$INDEX'"}]'
# Add to AKS approved list
az aks update \
--resource-group $AZURE_RG \
--name $AZURE_AKS \
--api-server-authorized-ip-ranges $(AZURE_RANGE)
enabled: false
displayName: Remove Agent IP to the AKS Cluster (IPs)
The container section specifies the container image to use for this job.
The first task is designed to add the agent IP to the api-server-authorized-ip-ranges to allow access to the Kubernetes APIs. Then, add the agent IP to the to the loadBalancerSourceRanges in the NGINX controller. This will automatically update the “internal” Network Security Group to allow the agent IP to connect to the AKS Cluster.
The second task illustrate the “simple” usage of the sas-viya CLI.
The third task removes the agent address to the loadBalancerSourceRanges and the api-server-authorized-ip-ranges.
By leveraging container jobs in Azure DevOps, you're not just building and testing your code; you're ensuring consistency, improving security, and optimizing resource utilization throughout your development pipeline.
This instance demonstrates that we can leverage Azure DevOps' standard features, such as installing the kubectl tool, while also incorporating additional functionalities not inherently provided, like integrating the sas-viya CLI into the same pipeline.
In an upcoming post, I will detail the process of integrating with Azure KeyVault to authenticate and fully utilize the features of the sas-viya CLI.
If you have an interest in Azure DevOps and exploring various ways to integrate it with SAS Viya, I recommend checking out Bogdan Teleuca posts, including topics such as:
Find more articles from SAS Global Enablement and Learning here.
Build your skills. Make connections. Enjoy creative freedom. Maybe change the world. Registration is now open through August 30th. Visit the SAS Hackathon homepage.
Register today!Data Literacy is for all, even absolute beginners. Jump on board with this free e-learning and boost your career prospects.