Setting Up SAS Viya with GitLab CI/CD Pipelines: Step One

[#1]

The purpose of this post series is to guide you to configure GitLab CI/CD pipelines for effective operation with SAS Viya. The first step on this journey is to register a GitLab runner and define an executor. What is the role of the GitLab runner, what is an executor and why do you need one for triggering SAS Viya workloads? Read the post to find out more.

Post Series

In this post series you will learn how to:

• Register the GitLab Kubernetes Runner – the current post.
• Create Your Own Docker Image for SAS Viya.
• Use GitLab Pipelines to Execute Commands in SAS Viya, using the Docker image you built as the base for your executor.

Before diving in the details, let’s answer the following question:

What's the Objective Here?

Implementing GitLab CI/CD pipelines brings numerous benefits to SAS Viya operations. These include automating processes, code versioning, fostering collaboration in development, facilitating content transfer across various environments, storing as well as archiving artifacts.

GitLab CI/CD pipelines empower you to:

• Automate segments of a business process.
• Execute SAS and Python programs from a version-controlled GitLab repository.
• Import SAS packages from a JSON file within the same repository.
• Initiate a SAS Studio flow or execute a SAS job.
• Export SAS packages into JSON files.
• And accomplish much more.

Video

Take a look at the following video, in which a GitLab Kubernetes runner is registered and tested.

GitLab Concepts

After the video, let's familiarize ourselves with some GitLab concepts:

Select any image to see a larger version.
Mobile users: To view the images, select the "Full" version at the bottom of the page.

GitLab Job: This is the smallest component of a pipeline, which contains one or more commands that need to be executed, such as to run a SAS program in SAS Viya. The jobs and the commands are defined inside a file .gitlab-ci.yml.

GitLab Runner: This is an agent installed on a server different from the GitLab server. The GitLab runner receives instructions from the GitLab server on which jobs to run. Each runner must be registered with the GitLab server.

Executor: Each runner requires at least one executor. An executor is the environment where the job will be executed.

Thus, the crucial distinction is: The GitLab Kubernetes runner acts as the agent that oversees and delegates the tasks, whereas the Kubernetes executor is the environment where these tasks are actually carried out.

GitLab offers various executors that you can use depending on your needs. These include Shell, Kubernetes, Docker, SSH, VirtualBox, Parallels, and Docker Machine.

Executors

In this post we will focus on a Kubernetes executor. Why Kubernetes?

We utilize a Kubernetes runner and executor since SAS Viya is deployed within a Kubernetes cluster. The Kubernetes executor pod is temporary, existing only for the duration of the job execution. Utilizing a Kubernetes executor is advantageous in this context. The runner spawns an executor pod using a Docker image, which can be sourced from a private container registry linked to your GitLab project or a public one like Docker Hub.

Setting Up a GitLab Runner

There are five simple steps to create the GitLab runner:

1. Create the runner.
2. Store the runner authentication token.
3. Deploy and start the runner.
4. Verify the runner registration.
5. Run a test job.

1. Create the runner

In your GitLab repository, perform the following steps:

• Navigate to Settings -> CI/CD and expand the 'General pipelines' section.
• Under 'Git strategy', set the 'Git shallow clone' field to 0. This has been found by my SAS colleagues to optimally support pipeline functioning.
• Expand the 'Runners' section, switch to the 'Instance runners' tab and deactivate instance runners.
• Generate a new project runner. Select 'Linux', add a tag, enable 'Run untagged jobs', designate the runner name, and set the 'Maximum job timeout' to a value superior to 600. Finally, create the Runner.
• Enabling 'Run untagged jobs' permits the runner to execute all pipelines, not just those specifying the tag.
• With the above option disabled, if your aim is to execute certain pipelines on the runner, it's necessary to incorporate the tag within the pipeline code.

2. Store the runner authentication token

Copy the value of the runner authentication token and save it somewhere safe.

GITLABTOKEN=glrt-....

3. Deploy and start the runner

For clear workload separation, the runner will be deployed in a fresh Kubernetes namespace. Executors will operate as pods within this specifically assigned workspace.

kubectl create namespace gitlab

A values.yaml file allows you to customize some of the runner pod settings, such as pods resources, number of replicas. You can find the full list of variables values.yaml.

cd ~
tee values.yaml > /dev/null << EOF
## Configure resource requests and limits for the runner pods
resources:
  limits:
    memory: 1024Mi
    cpu: 750m
  requests:
    memory: 512Mi
    cpu: 500m
## How many runner pods to launch
replicas: 2
EOF

The official way of deploying a GitLab runner instance into your Kubernetes cluster is by using the gitlab-runner Helm chart. This chart configures a GitLab runner to run using the Kubernetes executor. For each new job it receives from GitLab CI/CD, it provisions a new pod within the specified namespace to run it.

GITLABTOKEN=glrt-....
helm repo add gitlab https://charts.gitlab.io
helm install \
     -f values.yaml \
     --namespace gitlab gitlab-sasviya-008 \
     --set gitlabUrl=https://gitlab.com/,runnerRegistrationToken=$GITLABTOKEN \
     --set rbac.create=true \
     --set runners.privileged=true \
     --set checkInterval=5 \
     --set nodeSelector."kubernetes\.io/role"=agent \
     gitlab/gitlab-runner

Explanation of the options:

• 'helm repo' - Adds the GitLab Chart to Helm.
• 'helm install' - Deploys the runner.
• '-f values.yaml' - Includes the settings defined in your custom 'values.yaml' file.
• '--namespace gitlab' - Specifies the Kubernetes namespace.
• 'gitlab-sasviya-008' - Links with the runner name we registered in the GitLab interface.
• 'gitlabUrl' - Points to the full URL where your GitLab project is hosted. In some organizations, GitLab can be deployed on a company subdomain.
• 'runnerRegistrationToken' - Indicates the token generated when you registered the runner. The token allows the runner to authenticate in GitLab.
• '--set rbac.create' - If your cluster has RBAC enabled, you can opt to have the chart create the service account for you.
• '--set runners.privileged' - Allows the GitLab Runner to run using privileged containers. This might be necessary if you need to use the Docker executable within your GitLab CI/CD jobs.
• '--set checkInterval' - Defines in seconds how often to check GitLab for new builds.
• '--set nodeSelector."kubernetes.io/role"=agent' - Labels the GitLab Runner pods.

For more general information about Helm, see Helm basics.

4. Verify the runner registration

To verify the registered runners, navigate to Settings -> CI/CD and click on 'Expand' in the runners section.

5. Run a test job

Create a new file .gitlab-ci.yml in your GitLab repository, and paste the following inside:

show-hello:  
  script:  
    - echo "hello world"  

Commit the changes with a comment. Go to Build -> Pipelines and check the status.

Conclusion

In this post, we've walked through the process of setting up a GitLab runner for your GitLab repository in a Kubernetes cluster. This is an essential first step towards seamless CI/CD operations with SAS Viya.

In the following steps, you'll construct a Docker image that your Kubernetes executor will use to send commands to SAS Viya and write a few SAS Viya CI/CD pipelines.

Thank you for your time!

Acknowledgements

Many thanks to @JanKostrubiec, @katarzynazuk, @larsarne, and @Neil-Griffin from whom I learned the most about DevOps with SAS Viya and GitLab.  

Thank you for your time reading this post. If you liked the post, give it a thumbs up! Please comment and tell us what you think about having conversations with your data. If you wish to get more information, please write me an email.

Find more articles from SAS Global Enablement and Learning here.