SAS Communities Library

We’re smarter together. Learn from this collection of community knowledge and add your expertise.
BookmarkSubscribeRSS Feed

Setting Up SAS Viya with GitLab CI/CD Pipelines: Step One

Started ‎07-05-2024 by
Modified ‎07-18-2024 by
Views 1,777

The purpose of this post series is to guide you to configure GitLab CI/CD pipelines for effective operation with SAS Viya. The first step on this journey is to register a GitLab runner and define an executor. What is the role of the GitLab runner, what is an executor and why do you need one for triggering SAS Viya workloads? Read the post to find out more.

 

 

Post Series

 

In this post series you will learn how to:

 

  • Register the GitLab Kubernetes Runner – the current post.
  • Create Your Own Docker Image for SAS Viya.
  • Use GitLab Pipelines to Execute Commands in SAS Viya, using the Docker image you built as the base for your executor.

 

Before diving in the details, let’s answer the following question:

 

 

What's the Objective Here?

 

Implementing GitLab CI/CD pipelines brings numerous benefits to SAS Viya operations. These include automating processes, code versioning, fostering collaboration in development, facilitating content transfer across various environments, storing as well as archiving artifacts.

 

GitLab CI/CD pipelines empower you to:

 

  • Automate segments of a business process.
  • Execute SAS and Python programs from a version-controlled GitLab repository.
  • Import SAS packages from a JSON file within the same repository.
  • Initiate a SAS Studio flow or execute a SAS job.
  • Export SAS packages into JSON files.
  • And accomplish much more.

 

 

Video

 

Take a look at the following video, in which a GitLab Kubernetes runner is registered and tested.

 

SAS_Viya_GitLab_CICD_Pipelines_Step_One_Register_Kubernetes_Runner.mp4
Video Player is loading.
Current Time 0:00
Duration 0:00
Loaded: 0%
Stream Type LIVE
Remaining Time 0:00
 
1x
    • Chapters
    • descriptions off, selected
    • captions off, selected
      (view in My Videos)

       

       

      GitLab Concepts

       

      After the video, let's familiarize ourselves with some GitLab concepts:

       

      01_BT_GitLab_Kubernetes_Runner_Executor-1024x571.png

      Select any image to see a larger version.
      Mobile users: To view the images, select the "Full" version at the bottom of the page.

       

       

      GitLab Job: This is the smallest component of a pipeline, which contains one or more commands that need to be executed, such as to run a SAS program in SAS Viya. The jobs and the commands are defined inside a file .gitlab-ci.yml.

       

      GitLab Runner: This is an agent installed on a server different from the GitLab server. The GitLab runner receives instructions from the GitLab server on which jobs to run. Each runner must be registered with the GitLab server.

       

      Executor: Each runner requires at least one executor. An executor is the environment where the job will be executed.

       

      Thus, the crucial distinction is: The GitLab Kubernetes runner acts as the agent that oversees and delegates the tasks, whereas the Kubernetes executor is the environment where these tasks are actually carried out.

       

      GitLab offers various executors that you can use depending on your needs. These include Shell, Kubernetes, Docker, SSH, VirtualBox, Parallels, and Docker Machine.

       

       

      Executors

       

      In this post we will focus on a Kubernetes executor. Why Kubernetes?

       

      We utilize a Kubernetes runner and executor since SAS Viya is deployed within a Kubernetes cluster. The Kubernetes executor pod is temporary, existing only for the duration of the job execution. Utilizing a Kubernetes executor is advantageous in this context. The runner spawns an executor pod using a Docker image, which can be sourced from a private container registry linked to your GitLab project or a public one like Docker Hub.

       

       

      Setting Up a GitLab Runner

       

      There are five simple steps to create the GitLab runner:

       

      1. Create the runner.
      2. Store the runner authentication token.
      3. Deploy and start the runner.
      4. Verify the runner registration.
      5. Run a test job.

       

       

      1. Create the runner

       

      In your GitLab repository, perform the following steps:

       

      • Navigate to Settings -> CI/CD and expand the 'General pipelines' section.
      • Under 'Git strategy', set the 'Git shallow clone' field to 0. This has been found by my SAS colleagues to optimally support pipeline functioning.
      • Expand the 'Runners' section, switch to the 'Instance runners' tab and deactivate instance runners.
      • Generate a new project runner. Select 'Linux', add a tag, enable 'Run untagged jobs', designate the runner name, and set the 'Maximum job timeout' to a value superior to 600. Finally, create the Runner.
      • Enabling 'Run untagged jobs' permits the runner to execute all pipelines, not just those specifying the tag.
      • With the above option disabled, if your aim is to execute certain pipelines on the runner, it's necessary to incorporate the tag within the pipeline code.

       

       

      2. Store the runner authentication token

       

      Copy the value of the runner authentication token and save it somewhere safe.

       

      GITLABTOKEN=glrt-....

       

       

      3. Deploy and start the runner

       

      For clear workload separation, the runner will be deployed in a fresh Kubernetes namespace. Executors will operate as pods within this specifically assigned workspace.

       

      kubectl create namespace gitlab

       

      A values.yaml file allows you to customize some of the runner pod settings, such as pods resources, number of replicas. You can find the full list of variables values.yaml.

       

      cd ~
      tee values.yaml > /dev/null << EOF
      ## Configure resource requests and limits for the runner pods
      resources:
        limits:
          memory: 1024Mi
          cpu: 750m
        requests:
          memory: 512Mi
          cpu: 500m
      ## How many runner pods to launch
      replicas: 2
      EOF

       

      The official way of deploying a GitLab runner instance into your Kubernetes cluster is by using the gitlab-runner Helm chart. This chart configures a GitLab runner to run using the Kubernetes executor. For each new job it receives from GitLab CI/CD, it provisions a new pod within the specified namespace to run it.

       

      GITLABTOKEN=glrt-....
      helm repo add gitlab https://charts.gitlab.io
      helm install \
           -f values.yaml \
           --namespace gitlab gitlab-sasviya-008 \
           --set gitlabUrl=https://gitlab.com/,runnerRegistrationToken=$GITLABTOKEN \
           --set rbac.create=true \
           --set runners.privileged=true \
           --set checkInterval=5 \
           --set nodeSelector."kubernetes\.io/role"=agent \
           gitlab/gitlab-runner

       

      Explanation of the options:

       

      • 'helm repo' - Adds the GitLab Chart to Helm.
      • 'helm install' - Deploys the runner.
      • '-f values.yaml' - Includes the settings defined in your custom 'values.yaml' file.
      • '--namespace gitlab' - Specifies the Kubernetes namespace.
      • 'gitlab-sasviya-008' - Links with the runner name we registered in the GitLab interface.
      • 'gitlabUrl' - Points to the full URL where your GitLab project is hosted. In some organizations, GitLab can be deployed on a company subdomain.
      • 'runnerRegistrationToken' - Indicates the token generated when you registered the runner. The token allows the runner to authenticate in GitLab.
      • '--set rbac.create' - If your cluster has RBAC enabled, you can opt to have the chart create the service account for you.
      • '--set runners.privileged' - Allows the GitLab Runner to run using privileged containers. This might be necessary if you need to use the Docker executable within your GitLab CI/CD jobs.
      • '--set checkInterval' - Defines in seconds how often to check GitLab for new builds.
      • '--set nodeSelector."kubernetes.io/role"=agent' - Labels the GitLab Runner pods.

       

      For more general information about Helm, see Helm basics.

       

       

      4. Verify the runner registration

       

      To verify the registered runners, navigate to Settings -> CI/CD and click on 'Expand' in the runners section.

       

       

      5. Run a test job

       

      Create a new file .gitlab-ci.yml in your GitLab repository, and paste the following inside:

       

      show-hello:  
        script:  
          - echo "hello world"  
      

       

      Commit the changes with a comment. Go to Build -> Pipelines and check the status.

       

       

      Conclusion

       

      In this post, we've walked through the process of setting up a GitLab runner for your GitLab repository in a Kubernetes cluster. This is an essential first step towards seamless CI/CD operations with SAS Viya.

       

      In the following steps, you'll construct a Docker image that your Kubernetes executor will use to send commands to SAS Viya and write a few SAS Viya CI/CD pipelines.

       

      Thank you for your time!

       

       

      Acknowledgements

       

      Many thanks to @JanKostrubiec@katarzynazuk@larsarne, and @Neil-Griffin from whom I learned the most about DevOps with SAS Viya and GitLab.  

       

      Thank you for your time reading this post. If you liked the post, give it a thumbs up! Please comment and tell us what you think about having conversations with your data. If you wish to get more information, please write me an email.

       

       

      Find more articles from SAS Global Enablement and Learning here.

      Version history
      Last update:
      ‎07-18-2024 05:49 PM
      Updated by:
      Contributors

      sas-innovate-white.png

      Our biggest data and AI event of the year.

      Don’t miss the livestream kicking off May 7. It’s free. It’s easy. And it’s the best seat in the house.

      Join us virtually with our complimentary SAS Innovate Digital Pass. Watch live or on-demand in multiple languages, with translations available to help you get the most out of every session.

       

      Register now!

      SAS AI and Machine Learning Courses

      The rapid growth of AI technologies is driving an AI skills gap and demand for AI talent. Ready to grow your AI literacy? SAS offers free ways to get started for beginners, business leaders, and analytics professionals of all skill levels. Your future self will thank you.

      Get started

      Article Tags