A Practical Guide to Argo CD Part 3: Customizing Argo CD using Custom Tools and Secrets

[#1]

Introduction

Welcome back to the third and final installment of our blog series on ArgoCD and GitOps for SAS Viya (here are the other parts in case you missed them: Introduction and Core Concepts and Granular SAS Viya Deployments using Argo CD). This time, we’ll focus on a few practical details that make the setup more usable day to day, including custom kustomize tooling, sync control, and handling sensitive data with the External Secrets Operator.

As you already know, the purpose of ArgoCD is to synchronize your application with an external source, usually a Git repository. While CI tools like Jenkins, Tekton or GitHub Actions focus on building and testing, ArgoCD puts the emphasis on deploying and reconciling. In it’s most simple form, you’d upload your YAML manifests to a Git repository and create an ArgoCD Application definition, which tells ArgoCD to read in the manifests and deploy (“sync”) them as-is to a selected namespace on the Kubernetes cluster. Obviously, this is not quite enough for a deployment of SAS Viya, since SAS is not shipping ready-to-use YAML manifests but rather require that you use the kustomize tool to generate a final manifest (usually called site.yaml) based on templates provided by SAS and custom-written patches that contain the site-specific details needed for your site (certificates, hostnames etc.).

There are a couple of options to choose from at this point. You could run the kustomize command locally and then upload the final manifest to the Git repository or you could use a CI/CD toolchain, e.g. a CI tool like Tekton for this first step. And finally, there is another option: why not let ArgoCD do it for you?

ArgoCD ships with a built-in kustomize and if it detects a file called “kustomization.yaml” in your Git repository, it assumes that it needs to run the Git resources through kustomize first, before syncing them to Kubernetes.

In this blog, I’d like to describe how to use this approach for deploying SAS Viya. I’ll also discuss a few extra items, just to make things a little more interesting …

• We want to make sure to use the exact kustomize version which is required by SAS Viya.
• We need to be in control about which YAML objects we’re syncing – a few objects in site.yaml are not supposed to be deployed to the cluster without need.
• We don’t want to expose sensitive information like passwords, certificate keys or the SAS license to the Git repository. Rather, we’d like to keep these bits in an external secret database and use an additional tool to inject them directly into our namespace. In this blog, I’ll use the External Secrets Operator (ESO) for this purpose.

Adding kustomize v5.4.3 as a custom tool to ArgoCD

At the time of writing this blog, SAS Viya required the kustomize version 5.4.3 for building the main manifest. Chances are high, that the bundled kustomize in your ArgoCD deployment is newer than this. Here’s how to check:

$ kubectl -n argocd get deploy argocd-repo-server -o yaml | \
  yq '.metadata.labels | .["app.kubernetes.io/version"]'
v3.3.7

$ kubectl -n argocd exec deploy/argocd-repo-server \
  -c repo-server -- kustomize version
v5.8.1​

As you can see, my ArgoCD test system (v3.3.7) ships with kustomize v5.8.1. This might be ok for building the SAS manifests, but it is not guaranteed to work and errors caused by kustomize version mismatches are hard to debug and realistically can’t be fixed on your end.

Luckily, there is an easy way to add the kustomize version we need to ArgoCD. In short, you download the binary and store it on a volume that ArgoCD can access, then let ArgoCD know about the new “custom tool” and finally, in your Application definition, refer to this custom tool. As an added benefit of this approach, custom tools do not override the defaults, i.e. version 5.8.1 in my case will still be available and be used as the default unless I explicitly request version 5.4.3 for the SAS builds.

As you can see in the picture below, you could either store the custom kustomize binary on a shared volume (RWX) which is mounted to ArgoCD’s repo-server pod (option A). Alternatively, you can add an init-container to the repo-server pod and download the binary to an emptyDir volume (option B).

I’ve attached a helm-values YAML file to this blog that shows how to implement option B).

To let ArgoCD know about your custom tool, you need to add an entry to the main ConfigMap, argocd-cm. This entry has the form:

kustomize.path.<version string>=<path to binary>

So, assuming we want version 5.4.3, the filename of the binary is kustomize_5_4_3 and it is stored in a volume which is mapped to the repo-server pod at /custom-tools, the entry in the ConfigMap should looke like this:

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
[...]
data:
[...]
  kustomize.path.v5.4.3: /custom-tools/kustomize_5_4_3​

With these preparation steps in place, you can now refer to your custom tool from your ArgoCD Application:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
[...]

spec:
  source:
    kustomize:
      version: v5.4.3
[...]​

One word of caution: the version string you use in the Application needs to match exactly the version string used for the key in the ConfigMap.

Excluding unwanted objects from ArgoCD syncs

The result of the kustomize build process is a single manifest file which is usually called site.yaml. When deploying manually, you would usually sequentially run 4 kubectl apply commands, where each command filters out a subset of the objects contained in the site.yaml and submits them:

kubectl apply --selector="sas.com/admin=cluster-api" ...
kubectl apply --selector="sas.com/admin=cluster-wide" ...
kubectl apply --selector="sas.com/admin=cluster-local" ...
kubectl apply --selector="sas.com/admin=namespace" ...

Here’s a command that lets you review which object has which label:

$ yq -N -r '
  select(.kind and .metadata.name) |
  .metadata.name + "\t" +
  (.metadata.labels."sas.com/admin" // "(missing)")
' site.yaml | column -t

casdeployments.viya.sas.com                      cluster-api
dataservers.webinfdsvr.sas.com                   cluster-api
opendistroclusters.opendistro.sas.com            cluster-api
sas-analytics-events                             cluster-wide
sas-analytics-execution                          cluster-wide
[...]

ArgoCD neither honors this sequence nor the object labels used for filtering, but luckily this is not required – Kubernetes will sort it out for you. However, there is one gotcha – a few objects in site.yaml (see here for a list) are not labelled with either of these 4 strings. Therefore, they would be ignored in a manual deployment, but they will be synced by ArgoCD.

 You could either add transformer patches to your kustomization.yaml to suppress the generation of these objects, or you could make ArgoCD drop them during the build. The benefit of the latter approach is that you do not have to maintain a separate kustomization.yaml version for ArgoCD. Here’s how you can do that in your Application:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
[...]

spec:
  source:
    kustomize:
      version: v5.4.3
      patches:
        - target:
            kind: ConfigMap
            labelSelector: "sas.com/admin in (none)"
          patch: |
            apiVersion: v1
            kind: ConfigMap
            $patch: delete
            metadata:
              name: ignored
        - target:
            kind: Job
            labelSelector: "sas.com/admin in (none)"
          patch: |
            apiVersion: batch/v1
            kind: Job
            $patch: delete
            metadata:
              name: ignored

Avoid exposing sensitive information to your Git repository

Like elephants (and unlike humans), Git repositories never forget – which can quickly turn into a major security issue. Any SAS deployment needs to handle sensitive information, for example certificate keys, the SAS license file, connection information for the authentication provider or databases and more. This information is usually kept in plain text files and will be processed by kustomize to generate Kubernetes-native Secrets.

This creates a problem when using ArgoCD since the most common workflow involves a Git repository as a source, which would then need to store the sensitive information as well. Luckily, there are options available to avoid this, for example Sealed Secrets or the External Secrets Operator (ESO) frameworks. In this section, I’ll dive into the ESO a bit deeper.

The key element of the ESO architecture is that it externalizes the sensitive information and stores it in a secret management system outside Kubernetes. This changes your typical workflow as shown in the picture below:

While in case A) the sensitive information is part of your deployment assets, you would store this information separately in an external secret management system (like HashiCorp Vault, AWS Secrets Manager, etc.) for case B). The ESO basically acts as a bridge: it periodically pulls secrets from that external system and syncs them to your namespace as Kubernetes-native Secret objects. You define what you want via custom resources (ExternalSecret, SecretStore), and ESO takes care of fetching and keeping things in sync. Here’s a picture showing how the ESO works from a Kubernetes perspective:

  

From a deployment perspective this means that your kustomize patches will not contain any sensitive information and therefore you can safely push them to the Git repository. However, you do need to create the “placeholder” objects (the ExternalSecret custom resource) so that the ESO understands that it needs to take action.

Kubernetes Secrets vs ExternalSecrets

ExternalSecrets are custom additions to Kubernetes, introduced with the ESO. An ExternalSecret tells the ESO about the source and the target of the requested operation:

1. Where to look for the real data (the SecretStore, i.e. the external key vault)
2. Which key to read in that external system
3. Where to create the Secret, i.e. the target namespace, and its name

Let’s see how the ESO “translate” an ExternalSecret into a Secret (the below picture is simplified):

As you can see, the ExternalSecret contains all information bits needed to create the Secret (name, type, …). On top of that, it also contains information related to the external secret store.

The external secret store

ESO can pull secrets from a wide range of external stores - cloud services like AWS Secrets Manager, Azure Key Vault, and Google Secret Manager, as well as tools like Vault, IBM Secrets Manager, CyberArk, and others. It basically abstracts all of them behind the same SecretStore concept, so Kubernetes doesn’t care where the secret actually lives. For this blog, I’ve chosen OpenBao which is an open-source, community-driven fork of HashiCorp Vault that remains API-compatible.

Installing and preparing the OpenBao vault is outside of this blog, but here are 2 examples of how you could add keys to the vault that will later be turned into a Secret needed by SAS:

$ bao kv put -mount=secret viya-prod/postgres-user-env \
  username=dbmsowner \
  password='SecretPassword'
$ bao kv put -mount=secret viya-prod/sas-license \
  SAS_LICENSE=$(cat ~/SAS_license.jwt)

Turning Generated Secrets into ExternalSecrets with a kustomize plugin

Introducing the ESO to a deployment of SAS Viya adds a challenge: some of the Secrets you want to replace by ExternalSecrets are actually being generated during the kustomize process (see the secretGenerator section in your kustomization.yaml), so you can’t replace them easily.

One solution would be to use a post-build step (i.e. you’d have to rewrite parts of the site.yaml), however that would not fit nicely with using ArgoCD. For this blog, I’d like to describe a different approach which can be directly embedded into the kustomize build process and at the same time avoids making major changes to the kustomization.yaml.

For this approach we need to extend the functionality of kustomize through the help of a plugin. Kustomize plugins are built on the Kubernetes Resource Model, meaning they take Kubernetes-style YAML resources as input and return modified or generated resources in the same format. They can be implemented in essentially any language and are either packaged as container images or exposed as local executables, since kustomize only requires them to follow the KRM function contract for input/output handling. In practice, they are referenced via a config.kubernetes.io/function annotation, which tells kustomize how to execute them (for example via exec with a local binary or container with an image).

So, in the context of this blog, we need a plugin that takes a Secret and replaces it with a ExternalSecret. I’ve added an implementation of a plugin capable of doing this to this blog as an example. I won’t go into the implementation details, but the important bits are these:

• The plugin is added as a part of your usual build manifests. It is referred to in the kustomization.yaml as an additional YAML manifest in the site-config folder.
• This manifest tells kustomize where to find the external binary it should execute during the build process.
• There is no need for additional changes to the kustomization.yaml. All Secrets are created “as usual” during the kustomize build, but before they are written to the output manifest (site.yaml), the plugin will replace them with ExternalSecrets.
• Of course, you’d just add dummy values for these Secrets and not the real sensitive data ... You do want the secrets to be created after all (but their content is irrelevant).

The following bits are based on the example plugin I attached to the blog. This could be the reference to the plugin's configuration YAML manifest in your kustomization.yaml:

transformers:
(...)
- site-config/s2es-plugin/s2es-config.yaml    # kustomize external plugin for ESO

Where site-config/s2es/ has this content:

$viya-deploy
└── site-config
    └── s2es-plugin
        ├── s2es-config.yaml   # the configuration file
        └── s2es               # the binary (compiled Go code)

And the configuration of my plugin (s2es-config.yaml)  looks like this:

apiVersion: example.com/v1
kind: Secret2ES
metadata:
  name: secret2es
  annotations:
    # this is the important bit that tells kustomize where to find the plugin
    config.kubernetes.io/function: |
      exec:
        path: site-config/s2es-plugin/s2es
# this information is needed by the plugin for creating the ExternalSecrets
storeRefName: openbao
storeRefKind: ClusterSecretStore
remoteKeyPrefix: viya/db
refreshInterval: "1h"
creationPolicy: Owner
# this is a way to select the secrets you want to replace
# (a filter based on the name of the secret)
selectedSecrets:
  - sas-image-pull-secrets
  - sas-license
  - postgres-user-env

The plugin lets you pick the Secrets you want to protect. In the example above, I chose the image pull secret, the SAS license and the connection details for the external PostGreSQL database.

And that’s it? Well, almost ... There is one additional step: by default, kustomize ignores plugins. If you need to use one, you run the kustomize with additional flags. For a manual kustomize run, this would be the command-line you need:

kustomize build --enable-alpha-plugins --enable-exec -o site.yaml .

 Since we want to add these flags to the kustomize used by ArgoCD, we need to change the ConfigMap, argocd-cm, again and define them as additional build options:

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
[...]
data:
[...]
    kustomize.path.v5.4.3: /custom-tools/kustomize_5_4_3
    kustomize.buildOptions: "--enable-alpha-plugins --enable-exec"

If everything is in place, you can check the ArgoCD logs to confirm that the correct version of kustomize including these build options is used:

time="2026-04-30T16:37:52Z" level=info msg="/custom-tools/kustomize_5_4_3 build /tmp/_argocd-repo/(...)/manifests --enable-exec --enable-alpha-plugins" dir=/tmp/_argocd-repo/(...) execID=b781e

Conclusion

In this final post, we looked at a few practical details that make an Argo CD-based SAS Viya setup easier to run day to day: using the right kustomize version, controlling what gets synced, and keeping sensitive data out of Git with the External Secrets Operator.

Taken together, the three posts in this series move from the GitOps and Argo CD foundations, to a responsibility-based deployment model for SAS Viya, and finally to the operational details that make that model practical in real environments.

We hope you enjoyed this blog series and found it useful for your own Argo CD and SAS Viya deployments and please feel free to reach out in the comments if you have any questions or feedback.