Script to add statements such as lockdown paths and librefs to SAS compute service autoexec

[#1]

How would you add a few lines of code to your compute service autoexec from your post deployment automation scripts, if they aren't in the autoexec already?

Adding SAS code to a compute service or compute context autoexec is very easy to do interactively using SAS Environment Manager. If you only need to add a bit of SAS code to an autoexec code block once, then using Environment manager is definitely the quickest and easiest way to do it.

However, there are situations such as automating post deployment configuration for SAS Viya environments you design for someone else to deploy on demand, or which you will deploy over and over again, where only a scripted method will do.

In this post I'll present a custom bash function called add_code_to_compute_service_autoexec() which adds one or more lines of SAS code to the existing autoexec code block in your compute service, if the new code lines are not already part of the existing compute service autoexec.

add_code_to_compute_service_autoexec() uses the SAS Viya Command-Line Interface, and of course the wonderful JSON query utility jq. (jq is currently one of my favourite toys).

Update the Compute Service autoexec from a bash script

Define add_code_to_compute_service_autoexec()

The add_code_to_compute_service_autoexec() bash function below is idempotent: it checks whether the 'new' code is in fact already in the autoexec code block, verbatim, and it will only append the 'new' code if it is not already part of the existing code. This isn't necessarily a big 'wow' feature; your automation probably only runs once. But it's good practice.

You could use this function in your post deployment SAS Viya administration scripts to add paths to the SAS programming runtime lockdown paths list - a question from a colleague similar to the first sentence of this post prompted me to write the function and the post.

But you could also use the function to add any other one or more lines of SAS code to the autoexec. You could use it to add code to define a SAS macro that is commonly used in your organization, or to add a few libname statements or some SAS options statements that your users need immediately after deployment, and you want to ensure are in the autoexec just once, not repeated. You can even do all of those in one call.

Here is the bash script file defining the function add_code_to_compute_service_autoexec():

#!/bin/bash

###############################################################################
# Define function to add content of a text file to global compute service autoexec
# Usage: add_code_to_compute_service_autoexec ${file_containing_compute_autoexec_code_to_add}
###############################################################################

add_code_to_compute_service_autoexec ()
{
  # Delete any old working files from previous runs: /tmp/compute_*.json, /tmp/compute_*.txt
  rm -f /tmp/compute_*.json /tmp/compute_*.txt

  # Test that the input parameter is populated and the file path contains exists
  file_code_to_add=${1}
  if [ -f ${file_code_to_add} ]; then
    echo "File \"${file_code_to_add}\" exists"

    # Get a JSON file representing the whole of the current (global) compute service configuration instance, with all its sub-instances. This is a fairly large JSON document.
    sas-viya --output fulljson configuration configurations download -d sas.compute.server -s compute > /tmp/compute_config.json
    # cat /tmp/compute_config.json
    
    # Delete some of the parts of compute_config.json we won't need, 
    # including config instances other than the autoexec one.
    # (It may still contains some parts we don't need, but this is clean enough to be readable.)
    jq -r 'del(.items[] | select(.name != "autoexec_code"))' /tmp/compute_config.json | jq -r 'del(.accept, .count, .start, .limit, .links, .items[].links)' > /tmp/compute_autoexec_config.json
    # cat /tmp/compute_autoexec_config.json
    
    # Extract the current contents of the autoexec block from compute_autoexec_config.json
    # and save it to a file. This file contains a \n for each newline.
    jq -r '.items[0].contents' /tmp/compute_autoexec_config.json > /tmp/compute_autoexec_code_multiline.txt
    # cat /tmp/compute_autoexec_code_multiline.txt
    
    # Convert the multiline content of ${file_code_to_add} to a single line
    # with a \n for each newline, for use in the test below to see if it is already in the existing autoexec.
    cat ${file_code_to_add} | sed -n 'H;${x;s/^\n//;s/\n/\\n/g;p}' > /tmp/compute_autoexec_code_to_add_oneline.txt
    # cat /tmp/compute_autoexec_code_to_add_oneline.txt
    
    # Convert the multiline content of /tmp/compute_autoexec_code_multiline.txt to a single line
    # with a \n for each newline, for use in the test below.
    cat /tmp/compute_autoexec_code_multiline.txt | sed -n 'H;${x;s/^\n//;s/\n/\\n/g;p}' > /tmp/compute_autoexec_code_oneline.txt
    # cat /tmp/compute_autoexec_code_oneline.txt
    
    # Load strings containing the existing autoexec code and the candidate code to add to it in variables.
    compute_autoexec_code_oneline=$(cat /tmp/compute_autoexec_code_oneline.txt)
    compute_autoexec_code_to_add_oneline=$(cat /tmp/compute_autoexec_code_to_add_oneline.txt)
    # echo ${compute_autoexec_code_oneline}
    # echo ${compute_autoexec_code_to_add_oneline}
    
    # If compute_autoexec_code_oneline.txt does not already contain compute_autoexec_code_to_add_oneline.txt, add it. If it already contains that block of code verbatim, do nothing.
    if [[ ${compute_autoexec_code_oneline} == *"${compute_autoexec_code_to_add_oneline}"* ]]; then
      echo "The autoexec code block already contains that code. Nothing to do."
    else
      echo "The autoexec code block does not yet contain that code. Adding it."
    
      rm -f /tmp/compute_autoexec_code_multiline_final.txt
      cat /tmp/compute_autoexec_code_multiline.txt \
          ${file_code_to_add} > /tmp/compute_autoexec_code_multiline_final.txt
      # cat /tmp/compute_autoexec_code_multiline_final.txt
    
      # Replace the content value in compute_autoexec.json and save the resulting JSON in compute_autoexec_updated.json
      compute_autoexec_content=$(cat /tmp/compute_autoexec_code_multiline_final.txt)
      # echo $compute_autoexec_content
      jq -r --arg compute_autoexec_content "${compute_autoexec_content}" '.items[0].contents = $compute_autoexec_content' /tmp/compute_autoexec_config.json > /tmp/compute_autoexec_config_updated.json
      # cat /tmp/compute_autoexec_config_updated.json
    
      # Update the compute service configuration with this new autoexec section, leaving
      # the rest unchanged
      sas-viya configuration configurations update --file /tmp/compute_autoexec_config_updated.json
    fi
  else
    echo "File \"${file_code_to_add}\" not found"
  fi
}

Some notes:

• add_code_to_compute_service_autoexec()relies on the SAS Viya Command-line Interface (sas-viya), and expects to find it from the command sas-viya. So, before you use the function in this post, you must install and configure the CLI to connect to your SAS Viya deployment, and then authenticate as a user who is a SAS Administrator. This has been written about a lot, not least in these posts:
  • Gerry Nelson: SAS Viya Command-Line Interface for Administration (update)
  • Gerry Nelson: Overview of installing and using the SAS Viya containerized CLI
  • Stuart Rogers:  SAS Viya CLI with Group-Managed Service Accounts
• If the sas-viya executable or an alias or link to it is not in your path, you may need to modify the function to add the path to sas-viya in your environment, or provide it in an environment variable.
• This function also relies on jq: https://github.com/jqlang/jq.
• It deletes and creates and  a set of text files (containing JSON and plain text) in the /tmp directory, named like /tmp/compute_*.jsonand  /tmp/compute_*.txt. If this might cause an issue in your environment, feel free to modify the function as you see fit to store the temporary files somewhere else.
• There are quite a few commented-out cat and echo statements, which were useful to me during the function's development. I left them in the script in case they prove useful to you.

The function expects one (positional) parameter, which is the path to a file containing one or more lines of new SAS code, to be added to the global compute service autoexec.

Call add_code_to_compute_service_autoexec()

To call the function (after authenticating against the SAS Viya CLI and doing the other things in the bullet points just following the script above), simply create a text file containing the SAS code statements you wish to add (if they are not already in the compute service autoexec code block), and call the function passing the path to that text file as its only parameter:

#!/bin/bash

# Create a file containing the SAS code you want to add to an autoexec if it doesn't
# already contain this code.
tee /tmp/autoexec_code_to_add.txt > /dev/null << EOF
lockdown path='/shared_path';
libname shrdata '/shared_path/data';
EOF
# cat /tmp/autoexec_code_to_add.txt

add_code_to_compute_service_autoexec "/tmp/autoexec_code_to_add.txt"

Expected output:

File "/tmp/autoexec_code_to_add.txt" exists
The autoexec code block does not yet contain that code. Adding it.
"PATCH" "/configuration/configurations" complete from "/tmp/compute_autoexec_config_updated.json".

If you were to run the function a second time, with the same content in the file you pass as a parameter, you should instead see this:

File "/tmp/autoexec_code_to_add.txt" exists
The autoexec code block already contains that code. Nothing to do.

The example autoexec code statements shown above would make sense for a SAS Viya deployment in which you have mounted a volume into your compute server (and other SAS Programming Runtime) pods, which makes the filesystem path /shared_path available to processes running in those pods.

Gerry Nelson touched on how to do this in the section titled Mount NFS directories into Compute Pods, in his post SAS Viya: making user home-directories available to compute. It's quite a common deployment or post-deployment configuration task, and is described in the SAS Viya Platform Operations documentation, under Using Kubernetes Volumes for User Home Directories and Using NFS Server to Mount Home Directory. Those brief entries refer you to a README file in your deployment assets, here: $deploy/sas-bases/examples/sas-launcher/configure/README.md (for Markdown format) or here: $deploy/sas-bases/docs/configuration_settings_for_sas_launcher_service.htm (for HTML format).

If you are an expert in LOCKDOWN, you might know that the lockdown path statement in the example autoexec code above is not needed for the libname statement which follows, because libname statements in the global compute server autoexec and in compute context autoexec are run before the compute session enters the locked down state. However, adding the path /shared_path to the lockdown paths list would enable users to later run their own libname statements, like this for example:

libname projectx '/shared_path/projectx';

That libname statement would not be allowed in user code if /shared_path is not in the lockdown paths list. See the LOCKDOWN documentation in SAS Help Center, and this post for more on LOCKDOWN:

David Stern: How SAS Watchdog restricts filesystem access for Open Source programs

You could also add other code to your equivalent of my example /tmp/autoexec_code_to_add.txt. Anything that makes sense in an autoexec code block should work.

Be consistent with your indentation, trailing spaces and whitespace in general, if you care about the idempotency part of the function: that it won't add code that already exists in the compute service autoexec. The function doesn't do anything particularly clever about trimming white space, so for example if your compute service autoexec contains a line indented by one space, and you run the function to add the same line not indented, you should expect the 'duplicate' line to be added to the autoexec, and some warnings in the server logs when compute sessions start up.

What about automating updates for compute context autoexec code?

I also worked on a similar function to add lines to a compute context autoexec code block, as well as the one for the global compute service autoexec code block.

That proved more difficult, partly because the way autoexec lines are stored in the JSON representation of a compute context (a dictionary of strings, one string per autoexec line) is slightly different to how they are stored in the JSON for the compute service (a single string value with embedded newlines encoded as \n) which needs to be handled as appropriate for the idempotency test.

But it was more difficult mostly because there is not currently a SAS Viya Command-line Interface command like sas-viya compute contexts update. There is a command to create a new compute context from a JSON file, but not one to update an existing one. I reworked my idempotency check with the different representation of lines, but the second issue just made it a bit too much of a stretch for this post.

I think it can be done with an as-yet-to-be-written pyviyatool, similar to setcomputecontextattributes.py. But that one is towards the more complicated end of the scale among our pyviyatools and and I wanted to publish what I have so far, so that will have to wait for another time. Let me know in the replies if you'd like it sooner rather than later!

Update the Compute Service autoexec interactively, in SAS Environment Manager

If you only need to add code to your compute service autoexec once for a SAS Viya deployment, and you don't need to automate this task, do it interactively in SAS Environment Manager. This is documented in SAS Help Center, in the SAS Viya Platform Administration guide, Servers and Services, Programming Run-time Servers, SAS Compute: Server, Service and Contexts, in the How To section Edit Configuration Instances.

Sign in as an administrator and navigate to the Configuration page. (Users who are not members of the SAS Administrators custom group don't see the Configuration page). I find the easiest way to get to the configuration instance is to choose Definitions from the View dropdown, then type 'compute' in the search box, and select 'sas.compute.server', but there are other ways to get to the same set of configuration instances. Collapse the set of configuration instances on the right-hand side of the application page, so you can see them all.

Compute Service configuration instances, highlighting the autoexec code instance

Select any image to see a larger version.
Mobile users: To view the images, select the "Full" version at the bottom of the page.

Edit the configuration instance for Compute Service: autoexec_code. In the Edit sas.compute.server Configuration dialog, the contents field contains the autoexec code statements that are run as every compute session starts, regardless of which compute context it is running in:

Edit the global compute service autoexec code here

That is all you need to do if you don't need to script anything. But for those of us who look after automatically-deployed environments, I think the add_code_to_compute_service_autoexec() function, or something similar, will fill a gap in our toolset.

See you next time!

Find more articles from SAS Global Enablement and Learning here.