Updating to the Newest yq Syntax
Yq, a command-line YAML processor, is often used on SAS environments to make edits to files such as kustomize.yaml or other site configuration files. This powerful tool has received some recent upgrades, but it comes with a syntactical overhaul that may take some adjustments. This post will walk you through what yq is, what it does, and how to update from the previous version (yq v3) to the new yq v4.
So, what is yq?
As mentioned previously: yq is a command-line YAML processor. This phrase, broken down, just means yq processes (allows you to add, modify, read, delete) the content of .yaml files using commands on the linux terminal. A YAML file, for those unfamiliar, is a structured text file using key-value pairs with indentations to mark different "levels" of description. Below is an example of a YAML file about fruits:
Select any image to see a larger version.
Mobile users: To view the images, select the "Full" version at the bottom of the page.
We'll call this file "fruits.yaml". Here we see keys that contain more keys with values, as well as a key with two different values in a list. It can get more complicated once you get to deployment site configuration files, but this is a simple example. Really the "complication" of a YAML file is just how deep the keys within keys go, so this file will work for our purposes going forward.
What would we use yq for in fruits.yaml?
After first installing the yq command-line utility we could issue yq commands to change this file without ever having to manually open it. It's possible to do every type of modification to the file we would need using the command line alone, which is helpful for automation and scripting. You may have a script invoke yq to modify the color of our banana from "yellow" to "green". While other command-line based text editors could technically do that particular job, yq uniquely understands YAML syntax and makes it easier to make edits to one or more values at a time. Other general text editor utilities don't understand the tiered structure present in this file, which is what makes yq efficient at targeting values.
This is how yq sees this file, and how you'd reference to the values contained within using yq:
- apple.color describes the value under the key "color" which is under the key "apple".
- banana.color describes the value under the key "color" which is under the key "banana". It is okay to have "color" as an identical key under multiple fruits, this poses no problems in a YAML file.
- *.color would describe values held under the "color" key under both apple, banana, and orange. Modifying this might not be useful in this particular example, but imagine you stored some connection url information under multiple keys in a site environment's config YAML file and all of them needed to be changed, yq could do that sort of edit easily.
Updating from yq v3 to yq v4
Let's say you had a bunch of yq3 commands for building out and editing fruits.yaml. The new update to yq has changed how the utility functions... and you aren't sure where to start. We'll go over what changed in general with yq4, and then some specific yq3 -> yq4 command update examples using the new logic.
First, yq3 was incredibly specialized for a particular purpose: manipulating YAML files. This hasn't changed in yq4, but the philosophy has changed somewhat. Yq3 has specific commands for adding, deleting, editing, and reading values in YAML files. Yq4 takes this approach and simplifies it to an all-in-one edit command. Realistically an edit, deletion, and addition to some value in a YAML file is just a text edit. Deleting is the same as editing to be nothing, adding is just creating a new key and editing its value, and a standard edit just changes a value into a new value. Yq3 always output the result of a command to the console, allowing you to read what you had done, so why not bundle everything into the same command? Yq4 is that main fundamental change, with a few others sprinkled in.
Reading a value in yq3:
yq r fruits.yaml 'banana.color'
yq4 equivalent:
yq '.banana.color' fruits.yaml
Editing a value in yq3:
yq w fruits.yaml 'banana.color' green
yq4 equivalent:
yq '.banana.color = "green"' fruits.yaml
Adding a new key-value in yq3:
yq w fruits.yaml 'orange.texture' bumpy
yq4 equivalent:
yq '.orange.texture = "bumpy"' fruits.yaml
Deleting a key and its values in yq3:
yq d fruits.yaml 'banana.unbiased_tastiness_score'
(this deletes the key along with its corresponding value, 100)
yq4 equivalent:
yq 'del(.banana.unbiased_tastiness_score)' fruits.yaml
In yq4, invoking "yq" by default now invokes the all-in-one "eval" function. This eval function makes edits, additions, deletions, or lets you read a file or value based on whatever comes next in the command after "yq". Following the proposed changes (which are expressed in single quotes) is the file path, which tells yq where the file to be modified will be.
Why the changes?
Yq is already heavily based on jq, a popular command-line JSON processor, and this update aims to make the syntax between the two programs more similar. Several smaller features were dropped in favor of leveraging the more powerful already-existing linux terminal tools/strategies. For example, you used to be able to "validate" a yaml file using a custom yq command, and now you would just run a general linux command to check if your file throws errors. Comparing files used to be done with a custom "compare" command in yq, and now you would just use the "diff" linux utility to show the differences between two files. Because unneeded features were dropped, development time for the utility has then been able to be focused on improvements like allowing control over YAML metadata, letting users work with multiple files at a time, creating maps and arrays with YAML, and more. The latest iteration of yq is supposed to slot better into an automated workflow or pipeline and resemble the syntax of other familiar commands more clearly. Piping from STDIN to STDOUT (the linux terminal's standard input and standard output) has been properly defined, and the addition of an "eval" and "eval-all" function allows users to make both targeted and sweeping changes across one or more files with ease.
Read more about the changes and new syntax here.
Related Links:
Deploying SAS Viya 4 (course uses yq and explains the utility for use in deployments)
Find more articles from SAS Global Enablement and Learning here.
