To be truly CI/CD, you need to design and perform automated tests. Test automation reduces dramatically the time to release a feature by detecting errors early. Learn how to set up a simple DevOps pipeline in Jenkins, integrate a unit testing framework, SASUnit, and visualize the results.
Code development in continuous integration was introduced in the previous posts:
Time to focus on testing and more specifically, test automation. The topic can be very broad and we will break it into several parts:
This post presents the test results. In a test automation process, this is one of the last steps. It is maybe worthy you know what the end result is, before moving to more details.
To replicate the examples, you need the following technical components used: SAS 9, SAS 9 machine (Windows), Jenkins, Jenkins agent, SASUnit, Jenkins plugins: SASUnit, Blue Ocean, GitLab.
Select any image to see a larger version.
Mobile users: To view the images, select the "Full" version at the bottom of the page.
There are nice features that accelerate testing and centralize testing results. SASUnit is not a SAS supported product and you should treat like any open source project.
A Jenkins pipeline launches the unit testing in a build. I find Blue Ocean, a Jenkins plugin, quite useful to visualize the build and test results.
Jenkins connects to the agent, the Windows machine on which SAS 9 is running. The SASUnit installation folder is checked. SASUnit is the testing framework launching the tests.
Jenkins launches the unit tests in SASUnit, through the SASUnit Jenkins plugin.
After the tests ran successfully, the post-build steps will archive the artifacts. Tests will be evaluated and test results published as test reports.
A threshold qualifies the build as unstable. The threshold was defined in the 'Publish JUnit test result report'. You will see how you can use the attribute to chain pipelines, create dependencies and use the stable / unstable attributes as triggers in a future post.
Artifacts are various by-products generated during the process of software testing.
In the pipeline post build steps, the artifacts have been “grabbed” from the Jenkins pipeline workspace. As a reminder, the workspace is a folder on the Jenkins agent, the machine where SAS 9 is running.
Artifacts are generated at each build. They are archived, in case you need to look at them later, or compare different build versions. The artifacts are generated by SASUnit, when the unit tests are complete.
After the unit testing is complete you can consult the results. 55 test cases were executed, 3 failed and 52 have passed. A short description is available for the failed tests only. Unit tests have to be designed fast, not to slow the development pipeline.
Unit testing resembles more drag racing, than the 24 Hours of Le Mans.
The test results in the screen are fed from a junit.xml file (generated by SASUnit). The JUNIT.xml format is a standard way to report the test suite execution results. These XML files can be processed by programs like Jenkins to display results of the tests.
It is interesting to look at the structure of such a junit.xml file. You can open it from the Artifacts tab.
In Jenkins you can switch from Blue Ocean to the Classic view. A summary is then condensed on a single page.
In the Test Result, the tests, failed and successful are displayed.
In History, you can see the evolution in time of the number of failed tests, total tests executed and their execution time.
The testing framework may generate several reports and test documentation. SASUnit is generating a HTML report and test documentation, now saved in the Jenkins pipeline workspace.
The SASUnit report landing page:
The tests can be organized in SASUnit in a hierarchy: Test Scenarios ⭢ Test Cases ⭢ Assertions.
A test scenario is a group of test cases.
The scenario is successful if all the test cases pass.
A test case relates to a unit under test. The unit under test is the developed code. In DevOps world, you should think of unit as a single feature. The 1500 lines of code your colleague wrote and everyone loathes to maintain, contains many features.
A test case can have several assertions. An assertion is a Boolean expression at a specific point in a program which will be true unless there is a bug in the program. A test assertion is defined as an expression, which encapsulates some testable logic specified about the unit under test.
A test automation setup with SAS 9, SASUnit, a testing framework and Jenkins, an automation server can speed up testing, shorten development lifecycle and organize the test results in a logical, meaningful and transparent manner.
Navigate through the post series:
Thank you for your time reading this post. Please comment and share your experience with SAS and DevOps.
SAS is headed back to Vegas for an AI and analytics experience like no other! Whether you're an executive, manager, end user or SAS partner, SAS Innovate is designed for everyone on your team.
Interested in speaking? Content from our attendees is one of the reasons that makes SAS Innovate such a special event!
Data Literacy is for all, even absolute beginners. Jump on board with this free e-learning and boost your career prospects.