Automate SAS 9 Code Unit Tests in a DevOps Pipeline with Jenkins and SASUnit Part 1
To be truly CI/CD, you need to design and perform automated tests. Test automation reduces dramatically the time to release a feature by detecting errors early. Learn how to set up a simple DevOps pipeline in Jenkins, integrate a unit testing framework, SASUnit, and visualize the results.
Code development in continuous integration was introduced in the previous posts:
- DevOps Applied to SAS 9: SAS DI jobs, the Git Plug-in, GitLab and Jenkins
- DevOps Applied to SAS 9: SAS Code, GitLab and Jenkins
Time to focus on testing and more specifically, test automation. The topic can be very broad and we will break it into several parts:
- Setup SASUnit, a SAS 9 Testing Framework -> next post
- Setup Jenkins to work with SASUnit
- Add a new unit (SAS code) and test in SAS Unit
- Visualize tests results ⭢ this post
- Towards Continuous Testing
This post presents the test results. In a test automation process, this is one of the last steps. It is maybe worthy you know what the end result is, before moving to more details.
Summary
To replicate the examples, you need the following technical components used: SAS 9, SAS 9 machine (Windows), Jenkins, Jenkins agent, SASUnit, Jenkins plugins: SASUnit, Blue Ocean, GitLab.
Select any image to see a larger version.
Mobile users: To view the images, select the "Full" version at the bottom of the page.
SASUnit is a testing framework developed by a SAS partner. The code is open sourced on SourceForge and subject to a GNU General Public License. SASUnit integrates well with Jenkins through a plug-in.
There are nice features that accelerate testing and centralize testing results. SASUnit is not a SAS supported product and you should treat like any open source project.
Jenkins unit test pipeline
A Jenkins pipeline launches the unit testing in a build. I find Blue Ocean, a Jenkins plugin, quite useful to visualize the build and test results.
Step 1 Jenkins starts the build on the SAS 9 machine
Jenkins connects to the agent, the Windows machine on which SAS 9 is running. The SASUnit installation folder is checked. SASUnit is the testing framework launching the tests.
Step 2 Jenkins launches the unit tests through SASUnit
Jenkins launches the unit tests in SASUnit, through the SASUnit Jenkins plugin.
Step 3 Jenkins archives the artifacts and publishes the reports
After the tests ran successfully, the post-build steps will archive the artifacts. Tests will be evaluated and test results published as test reports.
A threshold qualifies the build as unstable. The threshold was defined in the 'Publish JUnit test result report'. You will see how you can use the attribute to chain pipelines, create dependencies and use the stable / unstable attributes as triggers in a future post.
Test artifacts
Artifacts are various by-products generated during the process of software testing.
In the pipeline post build steps, the artifacts have been “grabbed” from the Jenkins pipeline workspace. As a reminder, the workspace is a folder on the Jenkins agent, the machine where SAS 9 is running.
Artifacts are generated at each build. They are archived, in case you need to look at them later, or compare different build versions. The artifacts are generated by SASUnit, when the unit tests are complete.
Test results
After the unit testing is complete you can consult the results. 55 test cases were executed, 3 failed and 52 have passed. A short description is available for the failed tests only. Unit tests have to be designed fast, not to slow the development pipeline.
Unit testing resembles more drag racing, than the 24 Hours of Le Mans.
The test results in the screen are fed from a junit.xml file (generated by SASUnit). The JUNIT.xml format is a standard way to report the test suite execution results. These XML files can be processed by programs like Jenkins to display results of the tests.
It is interesting to look at the structure of such a junit.xml file. You can open it from the Artifacts tab.
Classic View
In Jenkins you can switch from Blue Ocean to the Classic view. A summary is then condensed on a single page.
In the Test Result, the tests, failed and successful are displayed.
In History, you can see the evolution in time of the number of failed tests, total tests executed and their execution time.
SASUnit HTML Report
The testing framework may generate several reports and test documentation. SASUnit is generating a HTML report and test documentation, now saved in the Jenkins pipeline workspace.
The SASUnit report landing page:
The tests can be organized in SASUnit in a hierarchy: Test Scenarios ⭢ Test Cases ⭢ Assertions.
A test scenario is a group of test cases.
The scenario is successful if all the test cases pass.
A test case relates to a unit under test. The unit under test is the developed code. In DevOps world, you should think of unit as a single feature. The 1500 lines of code your colleague wrote and everyone loathes to maintain, contains many features.
A test case can have several assertions. An assertion is a Boolean expression at a specific point in a program which will be true unless there is a bug in the program. A test assertion is defined as an expression, which encapsulates some testable logic specified about the unit under test.
Conclusions
A test automation setup with SAS 9, SASUnit, a testing framework and Jenkins, an automation server can speed up testing, shorten development lifecycle and organize the test results in a logical, meaningful and transparent manner.
Next
Navigate through the post series:
- Visualize tests results ⭢ this post
- Setup SASUnit, a SAS 9 Testing Framework ⭢ next post
- Setup Jenkins to work with SASUnit
- Add a new unit (SAS code) and test in SAS Unit
- Towards Continuous Testing
Recommended Resources
- DevOps Applied to SAS 9: SAS DI jobs, the Git Plug-in, GitLab and Jenkins
- DevOps Applied to SAS 9: SAS Code, GitLab and Jenkins
- SASUnit documentation.
- SASUnit Jenkins plug-in.
Thank you for your time reading this post. Please comment and share your experience with SAS and DevOps.
