kustomization.yaml – Where order is important (sometimes)

3 Likes

If you have deployed the new SAS Viya (2020.1 or later) on Kubernetes you have been exposed to the kustomization.yaml file used for customization of the environment. And depending on which components you have configured you may or may not have needed to consider the order in which customization items are placed in each of the respective sections. In this article we look at an issue you might encounter if you configure some of these items and don’t visit the Deployment Notes and reread the README.md files before your next update.

Background

The Global Enablement and Learning (GEL) team deploys and manages SAS Viya on long-running RACE reservations for a variety of workshops. In order to facilitate re-building these environments dynamically from scratch on a regular basis we have built a fair amount of automation.

This automation spans many facets of a SAS Viya environment, from ensuring the core software components are installed, such as Kubernetes and nginx, to creating NFS file systems, and ultimately deploying SAS Viya. Once SAS Viya is deployed additional automation performs more configuration, loads content, creates CASLIBs and loads data, as well as a myriad of other tasks to ensure the required pieces are in place.

An issue at Stable 2020.1.3

One customization task that was automated is the creation of a persistent volume for Python and R. This PV was created to make it available to the SAS Micro Analytic Service, CAS, SAS Compute, etc. Automation was developed to copy and configure the python-transformer.yaml, as well as update the kustomization.yaml to point to the configured file.

This automation was working as expected with Stable 2020.1.2. When Stable 2020.1.3 was released, deployed and tested, the configuration remained unchanged. However, there was one pod that never reached the Running state. That pod was the sas-microanalytic-score pod. It remained in the ImagePullBackOff status.

Select any image to see a larger version.
Mobile users: To view the images, select the "Full" version at the bottom of the page.

After some digging we discovered that it was failing to pull the image sas-microanalytic-score (see image below).

But if you look closely you will notice that a bit earlier it successfully pulled

cr.sas.com/viya-4-x64_oci_linux_2-docker/sas-microanalytic-score:2.42.29-20210326.1616794713240

If we take a closer look at the containers in the sas-microanalytic-score pod we see there are now two containers.

sas-microanalytic-score

viya4-mas-python-runner

But notice that the Image and Image ID for the viya4-mas-python-runner container are not fully qualified. In fact, there is no Image ID.

Since the Image for the viya4-mas-python-runner is sas-microanalytic-score and this is the same name as the Image for the sas-microanalytic-score container, one thought was to manually replace the existing Image with the one that was fully qualified for the sas-microanalytic-score container. Are you still with me? Essentially, we want to replace the image for the Python runner container with the image for the microanalytic score container so that they match.

Temporary fix

There are at least two ways to accomplish this. The easiest is to edit the configuration in Lens. Simply find the MAS pod in Lens and select it, then select the edit the configuration.

This opens the editor. Find “image:” for the sas-microanalytic-score container and copy the image property.

Then find the property for the viya4-mas-python-runner image and change from

to

Upon clicking “Save & Close” button, the changes will be saved and applied. And if we check the status, the pod is running.

The alternative temporary fix is to manually modify the manifest and essentially take the steps shown above, that is replace the image for the Python runner. Then apply the manifest. Keep in mind that the next build of the manifest will wipe out these changes. Of course, this is a test environment and the implications are minimal. Most likely we would not want to take these steps in a production environment, thus highlighting the need for test environment, even if it is a temporary environment.

Finding the true issue and resolving it

The prior steps are well and good for a one-time fix, but the same steps would have to be repeated if the manifest is applied again or if the pod was deleted it would revert to the original manifest.

After reviewing automation and other possibilities for the culprit, the next thing to look at was the original README.md for the Python transformer. When comparing the Stable 2020.1.3 instructions with the README.md for Stable 2020.1.2, it was discovered that the following line had changed (i.e. I relied on the automation and overlooked the Deployment Notes).

Note that the instructions now indicate that the python-transformer.yaml must precede the required 'sas-bases/overlays/required/transformers.yaml'.

Up until this point the automation used to configure and deploy SAS Viya was placing the Python transformer at the bottom of the transformers block in kustomization.yaml. It was subsequently changed to insert the Python transformer prior to the required transformers. The manifest was then rebuilt and applied, and sure enough the image and image ID subsequently resolved properly and the pod smoothly entered a running state. Changing the order allowed the build of site.yaml to generate the Image and Image ID for the Python side car container.

How many may be lurking?

This scenario raises the question about how those situations where order is important can be detected. Well the best method is to ensure that you visit the Deployment Notes to address customizations for each version update.

If you wanted to have a quick check, a simple method to find differences is to compare the README.md files using the diff or similar command for each transformer configuration of the current and new assets, similar to what was shown in the previous section. Of course, this requires that you keep a copy of the assets prior to applying an update.

In addition, you may want to compare the configuration files as they may have changed, and then use the most recent sas-bases configuration file for customization (i.e. don’t carry the old configuration to the new site-config).

Final Thoughts

In this use case we were performing a fresh deployment of Stable 2020.1.3 but using the site-config configuration files and customization from 2020.1.2 for automation. This isn’t really a case of updating software, but it still requires revisiting the deployment instructions and updating the configuration accordingly.

The bottom line is when performing a SAS Viya version update be sure to review the Deployment Notes and related README instructions for each existing customization. This is especially true if you have automation built to facilitate the deployment or update process as you want to make sure you are referencing the latest version of the assets. Failure to do so may lead you down the road of deployment failure and wasted time.

Deployment notes: SAS Help Center: SAS Viya: Deployment Notes

Find more articles from SAS Global Enablement and Learning here.