BookmarkSubscribeRSS Feed

SAS Viya Configuration Overwriter

Started ‎07-25-2024 by
Modified ‎07-22-2024 by
Views 2,354

SAS Viya Configuration Overwriter

 

SAS Viya has an extensive configuration system that allows you to modify the environment to your specific needs. These are stored as key-value pairs in the SAS Configuration Server. The Viya configuration can be managed in two main ways, through SAS Environment Manager and through the configuration plugin in the SAS Viya CLI. When managing multiple environments you may want to automate and standardize the configuration of an environment. The default way of achieving this is to use a sitedefault file to bulk-load configuration settings. Using the sitedefault file has its drawbacks though:

  • Configuration is only loaded upon starting Consul
  • Configuration in sitedefault will never overwrite changes made through SAS Environment Manager or the CLI
  • Configuration settings cannot be changed using sitedefault, only new settings can be added

So if you want to enforce that an environment is configured with a specific set of configuration settings you will need an alternative way to handle this.

To this end, I created a Kubernetes CronJob that uses the Viya CLI container image to automate the writing of configuration settings based on your existing sitedefault file. Because this CronJob uses the CLI and is independent of Consul, it removes the two limitations mentioned above. It forcefully overwrites the configuration in Consul so that it matches with the defined settings in the sitedefault file. As the process only uses SAS-provided and documented interfaces, it can be used in any environment.

 

The process is visually shown below:

Configuration Overwriter (1).png

 

The CronJob is deployed using an Azure DevOps pipeline that also converts the sitedefault file from YAML to JSON so that it can be parsed using the jq binary included in the SAS Viya CLI image. Although I used Azure DevOps to deploy it, the steps in the pipeline can easily be converted and used with other automation tools or scripts.

Once the sitedefault file is converted, the pipeline creates a configmap out of it. The only manual step is to register client credentials with which the Configuration Overwriter can access to Viya environment. Please ensure that SASAdministrators is added as an "authority" for the client credentials in order to allow the oauth client to change all configuration settings.

 

You can find the pipeline and CronJob definition attached to this post. I will highlight the critical parts of both below.

The pipeline uses Azure DevOps specific tasks de convert the sitedefault file and deploy it as a configmap, but essentially executes the following commands:

yq site-config/sitedefault.yaml -o json > sitedefault.json
kubectl create configmap sitedefault \
--from-file sitedefault.json \
--dry-run=client -o yaml > sitedefault_cm.yaml
kubectl apply -f sitedefault_cm.yaml

After this has been completed, it deploys a CronJob with the following job template:

jobTemplate:
    spec:
      template:
        spec:
          volumes:
          - name: sitedefault
            configMap:
              name: sitedefault

          containers:
            - name: configuration-overwriter
              image: <acr name>.azurecr.io/viya-4-x64_oci_linux_2-docker/sas-viya-cli:<cli version>
              command: ["/bin/bash","-c"]
              args:
              - |
#1              export OAUTH_TOKEN=$(curl -k -X POST 'https://<environment fqdn>/SASLogon/oauth/token' -H 'Content-Type: application/x-www-form-urlencoded' -d 'grant_type=client_credentials' -u $client_id:$client_secret | jq .access_token --raw-output)
#2              for service in $(cat /sitedefault/sitedefault.json | jq -r '.config | keys[]'); do
#3                for definition in $(cat /sitedefault/sitedefault.json | jq --arg service "$service" -r '.config[$service] | keys[]'); do
#4                  for setting in $(cat /sitedefault/sitedefault.json | jq --arg service "$service" --arg definition "$definition" -r '.config[$service][$definition] | keys[]'); do
                      if [ "$service" == "application" ]; then applied_service="GLOBAL"; else applied_service="$service"; fi
#5                      update_value=$(cat /sitedefault/sitedefault.json | jq --arg service "$service" --arg definition "$definition" --arg setting "$setting" '.config[$service][$definition][$setting]');
#6                      ./sas-viya --quiet configuration configurations download --definition-name $definition --service $applied_service --target ${definition}_${applied_service}.json
#7                    current_value=$(jq --arg setting "$setting" '(.items[0][$setting])' ${definition}_${applied_service}.json)
                      if [ "$update_value" == "$current_value" ];
                      then
                        echo "Not updating $applied_service definition $definition. Setting $setting is already configured as $update_value"
                      else
                        echo "Updating $applied_service definition $definition. Changing setting $setting from $current_value to $update_value"
                        jq --arg setting "$setting" --argjson update_value "$update_value" '(.items[0][$setting]) = $update_value' ${definition}_$applied_service.json > ${definition}_${applied_service}_update.json
                        ./sas-viya configuration configurations update --file ${definition}_${applied_service}_update.json
                      fi
                    done
                  done
                done
              env:
#8            - name: SAS_SERVICES_ENDPOINT
                value: "https://<environment fqdn>"
#9            - name: SAS_SKIP_OAUTH_EXPIRY_CHECK
                value: "true"
              envFrom:
#10           - secretRef:
                name: configuration-overwriter-secret

#11           volumeMounts:
              - name: sitedefault
                mountPath: /sitedefault

 

A little explanation on the highlighted lines in the code:

  1. Obtain an oauth token using client credentials
  2. Loop through all services defined in sitedefault
  3. Loop through all definitions for a particular service
  4. Loop through all settings for a particular definition
  5. Retrieve the value to set for that particular setting
  6. Download the current configuration definition for the service
  7. Retrieve the current value of the setting from the downloaded configuration
  8. We need to set SAS_SERVICE_ENDPOINT
  9. By default, SAS checks whether a user is logged on by looking at stored credentials. As we do not log on to the environment, but use OAUTH_TOKEN variable directly, we need to set SAS_SKIP_OAUTH_EXPIRY_CHECK.
  10. The client credentials required by the Configuration Overwriter should be stored in a secret within the SAS Viya namespace prior to deployment.
  11. Here we mount the converted sitedefault file so it can be read by the Configuration Overwriter.

I hope this utility proves helpful and allows you to improve the stability and consistency of your Viya environments. It allows for a quick and consistent way to apply configuration settings to your environment and makes it much easier to deploy the same set of configuration over multiple environment with no chance of user error.

Comments

Hi @JanStienstra 

 

So the "sas-viya configuration" CLI plug-in replaces the sas-bootstrap-config command that was in Viya 3.5?

It has the same equivalent capabilities as the sas-bootstrap-config command has?

 

Thanks

Hi @EyalGonen,


The sas-bootstrap-config utiltiy is still there in Viya 4, but it is embedded in the sas-consul pods.
The configuration CLI is available for both Viya 3.5 and Viya 4.

 

The sas-bootstrap-config utility is used to make changes directly in the key-value store that is consul.
The configuration CLI can be used to make these changes as well, but works via the configuration service.
This has the advantage that it does some checking on what you ask it to configure. i.e. are the settings you request it to add valid for a particular service or definition.

 

In short both can be used, but I prefer to use the configuration CLI as it performs some checks and it only requires access to the web applications, not to the underlying cluster and pods.

Thank you @JanStienstra  for the quick reply.

 

I have one more question: suppose I want to modify the value for a specific key, using sas-bootstrap-config I could accomplish that but it seems that with the CLI I need to prepare a JSON file to accomplish this (correct me if I am wrong). What is not clear to me is how to I know how to create this JSON file? The only way I think is to use the CLI option: "sas-viya configuration configurations download --service xxxx" to download the JSON, modify it and then use the CLI again to update like you did in your code in the blog. Is this the only way or is there some documentation/examples how to create the JSON without downloading it first?

Hi @EyalGonen,


The format of these JSON files is the same for all configuration instances.
However, it is the specific settings that you need to set what you need to discover. For that it is indeed recommended to download the existing configuration first and then make changes.

OK. One last question about this: let's say I download the JSON and it contains multiple key-values. I want to modify just one of those key-values. Can I "reduce" the JSON to keep only the key-value I need and drop all the key-values which I do not wish to modify?

(asking because in sas-bootstrap-config I could target a specific key-value)

 

For example. if I want to modify this key-value:

 

/opt/sas/viya/home/bin/sas-bootstrap-config kv write --force "config/identities/sas.identities.providers.ldap.user/emailAddress.work" "mail"

If this JSON equivalent? Will this work? I "reduced" the JSON to only the specific key-value I want to modify and dropped all other key-values which should remain as-is.

 

{
  "config": {
    "identities": {
      "sas.identities.providers.ldap.user": {
        "emailAddress.work": "mail"
      }
    }
  }
}

Very nice article Jan / @JanStienstra ! Thank you.

@JanStienstra  according to SAS documentation the sitedefault.yaml file, is used primarily for the bulk loading of configuration values. After the initial deployment, the sitedefault.yaml file cannot be used to modify an existing value and to deploy the software again. The sitedefault.yaml file can be used only to set new property values that have not already been set.

The question is, is the sitedefault.yaml used only at initial deployment or also later when new values have been added?

Hi @EyalGonen,

If you use the CLI directly, i.e. without using the script in this post, you need to provide the entire JSON again. So all other key-value pairs need to be included.

When you use the script, you can specify only the settings you want, as the updates into the JSON files is performed by the script and it will do the download and update for you (ensuring all existing values will also be in the file).

Hi @touwen_k,

The sitedefault.yaml can also be used later when adding new values, but like I mentioned it is only read upon starting Consul.

Hi @JanStienstra 

 

Thanks for all the replies!

 

Just one note: seems to me like your code can misbehave in a situation where a  updated service contains multiple configuration instances from the same definition for example the "compute" service has five configurations based on the "sas.compute.server" definition. So your code will only update the first instance (whichever is returned first by the CLI  download action). I assume this is intentional but better place some warning about this as this can potentially be destructive if not well understood and improperly used. Of course, correct me if I am wrong.

Hi @EyalGonen,

You are right that I did not consider this situation. I'm working on a new version of this script that will work for both Viya 3 and Viya 4, performs more input validation and will flag the situation you describe.
Thanks for your input!

Version history
Last update:
‎07-22-2024 10:05 AM
Updated by:
Contributors

SAS Innovate 2025: Register Now

Registration is now open for SAS Innovate 2025 , our biggest and most exciting global event of the year! Join us in Orlando, FL, May 6-9.
Sign up by Dec. 31 to get the 2024 rate of just $495.
Register now!

Free course: Data Literacy Essentials

Data Literacy is for all, even absolute beginners. Jump on board with this free e-learning  and boost your career prospects.

Get Started

Article Tags