DevOps for SAS Viya 3.5 Deployments: Configuring Gitlab-Triggered Builds in Jenkins

[#1]

Most likely you've heard the term Jenkins, but maybe you haven't had the time to dig into the ways it may benefit you as a SAS Viya installer, architect, administrator, or user. In brief, Jenkins is an open-source automation tool used heavily in software development. However, its capabilities are wide-ranging and can certainly be applied to many facets of a SAS Viya 3.5 deployment.

One of the commonly-used features of Jenkins is its ability to trigger builds based on an event or a schedule. For example, when configured to interact with Gitlab, you can trigger a build to execute each time an update to a Gitlab project occurs.  Assuming that example, let's take a look at what is required to trigger a Jenkins build and review some potential use cases.

Jenkins

Jenkins is an open-source automation server with many available plugins to extend functionality.  It is used by R&D and testing to automate building, deploying,  and testing throughout  the software development life cycle.

But don't be fooled into thinking that it is strictly for software development. There are many capabilities that enable it to be an effective tool to automate many SAS Viya 3.5 processes, both deployment and operations related.

If you've ever seen a screenshot or have logged on to the application, don't let the "classic" web interface fool you.  Behind that interface is a ton of functionality to automate your most-loved processes.

Select any image to see a larger version.
Mobile users: To view the images, select the "Full" version at the bottom of the page.

If you want to modernize the interface, you can check out the Blue Ocean interface. Check out the online doc for more details.

Let's review some key Jenkins terminology.

Plugins - Much of that functionality is the result of the plugins that interface with other technologies (e.g. Gitlab, Github, cloud providers, etc).  And lest I forget, it is extensible.  So you can enrich the functionality ever further if you wish by writing your own plugin.

Master - The central coordinating process/server which stores the configuration, provides and manages plugins, and renders the interface.

Project (Job) - At one point these terms were used interchangeably. However, the term "Job" has been deprecated.  Essentially projects are the core runnable tasks that are controlled and monitored by Jenkins.  Projects are created using the "New Item" menu and after selecting a type of project, you are presented with a series of configuration sections. The project can represent a single step or a complex combination of steps that run across multiple machines.

Build - A build is simply an execution or run of a project.

Artifact - Artifacts are files or objects created during a build.  Artifacts can be tossed at the end of a run or saved for future use.

Node - Nodes are the hosts/machines that are set up to run the project builds when run from a master.  In order for it to run tasks on a node, an agent must be defined on that machine.  After a node is registered to the master, the master then distributes work to the nodes.  Work can be directed using a variety of options.  The Master is also a node.

Workspace - This is a disposable directory on the node that is used as a working directory for the build.

Pipeline - A user-defined model of a continuous delivery pipeline.  Okay, that definition may not be too helpful.  A pipeline is a definition of a series of stages to be executed either serially or in parallel on one or more hosts. Stages may contain one or more steps.  For you mainframers, it reminds me of an uber-functional JCL with a web interface.  But I digress…

Stage - a grouping in a Pipeline that defines a subset of the entire pipeline

Step - a single task within a pipeline; a group of steps can be used to define a stage.  An example would be a single host command

For more terms, check out the Jenkins glossary.

Here is a simple architecture diagram of the Jenkins architecture (thanks Rob Collum).   Notice we can have agents on a variety of platforms and each agent needs a workspace to store its repositories and project content (artifacts).  If running a parallel pipeline, all three agents can be actively working at the same time.  Logs for each stage are made available to and stored on the Jenkins server (Master).  The Master can integrate with a variety of Git flavors as well as other software technologies.

In this example, we take advantage of Jenkins' Docker image.  Using Docker to initiate a Jenkins environment provides a fast path to quickly getting access to a Jenkins server.

Gitlab

I'll keep this section short.

Most likely you've dealt with one or more of the Git flavors in recent years.  Gitlab, and its competitors, are web-based DevOps lifecycle tools which provide a Git-repository manager for versioning, wiki capabilities, and issue-tracking along with various CI/CD pipeline features.  Gitlab operates using an open-source license.  A fair amount of overlap exists with the CI/CD pipeline features and Jenkins.  In this example, we use Gitlab purely as a versioning tool.

Triggers

A key feature in building automation in Jenkins is the ability to automatically kick-off (trigger) a project build as a result of an event, a request, or a predefined schedule.  One manifestation of this ability is configuring a Jenkins project to perform a task when a push to a Gitlab project is issued and completed.  For example, you have a YAML file that you are modifying in Gitlab and each time you modify it you want to perform syntax checker on the updated file (could be JSON, XML, etc). This is a simple example, but the point is to drive home the idea behind the example.  So let's walk through the steps to enable this ability.

Jenkins Key Components

In order to create the trigger build we need to ensure we have the required software so that Jenkins can integrate with a Gitlab repository.

• Jenkins software - probably the fastest way to get Jenkins installed and running is via a Docker image.  If you have Docker installed you could have Jenkins running within minutes.  Images can be found here.
• Gitlab instance - In many cases, this may already be available, you will just need to acquire an account that creates a repository or request authorization.  If a Gitlab instance isn't available, using Docker to deploy Gitlab is a fast way to get an instance running for your testing.
• Jenkins Plugins for Gitlab - Since in this example we will be interacting with Gitlab, there are specific plugins required.  If you use the standard Docker image, you will need two additional plugins.  To add plugins, go to Jenkins home => Manage Jenkins => Manage Plugins and then select the Available tab.
• Key plugins are Gitlab Plugin and Gitlab Hook Plugin.  If you want to build merge requests from Jenkins, include the Gitlab Merge Request Builder. Jenkins may have to restart to enable the plugins.  This takes a couple of minutes.  They will then show under the Installed tab

At this point you should have the key software pieces in place to begin configuration to perform a triggered build.

Trigger Project Configuration and Execution

To activate a triggered build, we will break this up into two phases, configuration and operation.

The following diagram highlights the sequence of actions required to configure a Jenkins project to be triggered when an update is pushed/merged/committed to a Gitlab project.  There are four major steps.

1. We first need to generate a personal Gitlab API Access Token that will be used by Jenkins to access the associated Gitlab project.  The access token is acquired via the Gitlab user's settings.  Once the token is generated, it should be added to Jenkins' credentials.
2. Next, we want to create a new project in Jenkins.  This can be a freestyle or pipeline project.  Within the project, we define what type of trigger we should initiate in the Jenkins job.  When the option, Build when a change is pushed to GitLab, is selected it provides the Gitlab webhook URL to be used in configuring Gitlab.  The webhook will look something like http://myjenkins.sas.com:8080/project/MyTriggerProject (not a real URL).  Note the other available triggers.
3. With the Jenkins Webhook URL return to Gitlab with the desired repository selected and select Settings => Webhooks.  Here you add the URL and define which events in Gitlab you want to create the trigger.  Trigger examples are Push, Merge, Comments, etc.  Once the webhook is added, you can test it to ensure it triggers as expected.
4. Now that we have the webhook configured in Gitlab, return to Jenkins and configure your Jenkins project.  There are many possibilities here.  Here we will present a very simplistic example with a few of the options.
   1. Define on which node you want the project to execute.  In the example, we execute on Node01.
   2. Add the Gitlab repository to the Jenkins project. This repository will then be cloned to the workspace of Node01.
   3. We defined the trigger in step 2
   4. Define what type of build action we want to execute when the trigger is initiated.  Some examples are to execute a script that is part of the repository or run a system process against a file or multiple files in the repository.
   At this point, the configuration is done and we are ready to begin testing (operation).

   

5. Depending on which Gitlab actions were configured for the Webhook, you want to generate that action.  For example, make a change to a file in the repository and commit the update which results in a Push.
6. The Gitlab action creates a call to the Webhook, which calls Jenkins with the trigger project URL.
7. The trigger build is initiated on the server, but the actual build was specified to run on Node01. So the required pieces from project are copied to the agent node to the defined workspace location.
8. Since we specified a Gitlab repository in the project, the build information requests that the Gitlab repository be cloned to the agent node.
9. With all the pieces in place, the build process executes on Node01 and the results and logs are returned to the server.

So there you have a simplistic view of the steps to configure Jenkins and Gitlab to initiate a trigger build job.

A Few Trigger Use Cases

Earlier we mentioned a very simple use case.  Assume we have the SAS Viya playbook stored in a Gitlab repository and we are planning a new deployment.  Most likely you will be making changes to the vars.yml file.  As you are making changes you could define a trigger build that will invoke a YAML syntax checker, such as YAMLlint, each time you update vars.yml to validate it.  This could save you time down the road with debugging changes to your deployment.

One key phase of a new deployment, or changes to an existing deployment, is to validate that services in a new or updated environment are functioning as expected. A trigger build could be established to be initiated after all services are running to verify the health of services and subsequently perform basic functionality testing.  Jenkins can be configured to retain the logs and output.  It captures response times as well. As a result the Jenkins user interface can provide a historical view of past validation tests and you can quickly identify executions that do not complete within the expected execution time.

Beyond these, it is up to your imagination to identify the many possibilities of creating trigger builds for a SAS Viya 3.5 deployment.

Final Thoughts

In this article, we've reviewed the basic process flow of triggering a Jenkins project build.  If you haven't been exposed to Jenkins before and your responsibilities include performing repetitive processes that would benefit from automation it may be worth your time to take a look at its capabilities.  With Gitlab and Jenkins Docker images you can fairly quickly have an environment running and begin trigger testing.

Thanks for reading!