SAS Viya Startup Sequencer
In this post, I will look at a new feature introduced in SAS Viya 2022.1.1, and enabled by default in stable 2023.01 the SAS Startup Sequencer. The components that comprise SAS Viya are designed to start in any order. This is because they take advantage of the built-in capabilities of Kubernetes to maintain the “desired state”. On startup, all Viya pods start at the same time and Kubernetes works to establish a “running” state for all the pods that make up the Viya deployment. This process continues until all Pods are running. As a result, SAS Viya can start without any built-in “SAS” logic in the startup process. However, as we know from experience with Viya 3.x, some SAS components cannot operate before the components they are dependent upon start. For example, all Viya Services will require the Configuration Server, the Infrastructure Data Server, and the Message Server to be started before they can begin to perform their function.
That is not a problem for Viya on Kubernetes. Services that cannot start will just keep retrying until the components they are dependent on are available and they can start successfully. That is all good stuff and very Kubernetes compliant. However, from an efficiency and resources perspective, it can be a problem. The constant restart attempts of pods can consume a large amount of CPU and Memory and slow the whole process down.
Introduced in Viya 2022.1.1, and enabled by default in 2023.01, and later the Startup Sequencer changes this behavior. The Startup Sequencer forces SAS Viya to start in an order that is more logical for SAS software and minimizes the usage of resources like memory and CPU.
How do we use it?
SAS Viya 2023.01 and later
In SAS Viya 2023.01 and later the start sequencer is enabled by default. No action is required to use it.
To provide full flexibility to the Viya Administrator an overlay is provided that will disable the functionality provided by the SAS Startup Sequencer. To disable the Startup Sequencer add <project-dir>/sas-bases/overlays/startup/disable-startup-transformer.yaml to the transformers block in your base kustomization.yaml ($deploy/kustomization.yaml) file. Ensure that disable-startup-transformer.yaml is listed after sas-bases/overlays/required/transformers.yaml.
...
transformers:
...
- sas-bases/overlays/required/transformers.yaml
- sas-bases/overlays/startup/disable-startup-transformer.yaml
SAS Viya 2022.1.1 to 2023.01
Prior to 2023.01 Startup Sequencer is NOT implemented by default and an overlay is supplied that optionally implements the Startup Sequencer.
The overlay is delivered in the sas-bases directory at <project-dir>/sas-bases/overlays/startup/ordered-startup-transformer.yaml and its use is documented in <project-dir>/sas-bases/examples/startup/README.md". The usage is simple, reference the overlay in the transformers section of your kustomization.yaml before the line that includes "- sas-bases/overlays/required/transformers.yaml".
...
transformers:
...
- sas-bases/overlays/startup/ordered-startup-transformer.yaml
- sas-bases/overlays/required/transformers.yaml
With this change made build and apply the manifest to implement the transform. If you do this step at initial deployment, then the startup sequence is respected when the software deploys. If you do this step post-deployment it will restart some of the pods and subsequently, the startup sequence is respected the next time you stop and start your Viya environment.
NOTE: if prior to 2023.01 you have implemented Startup Sequencer using the ordered-startup-transformer.yaml overlay, in 2023.02 and later it is no longer required. You can remove the line from your kustomization.yaml file. If you do not remove the line it will be ignored and have no negative impact on the deployment.
How does it work?
The transform adds an init container (start-sequencer) to the Viya deployments and stateful sets. An init container is a container that runs:
- when a pod starts
- to completion
- before the next container in a Pod starts.
The start-sequencer init container enforces the startup sequence by:
- Allowing the core services to start first, e.g., the Infrastructure Data Server, Configuration Server, CAS operator, Readiness service, etc.
- Pausing the startup of the other services when their start-sequencer init container runs until their dependent services are started.
The init container simply performs a sleep and waits for the availability of a dependent resource in the previous step. You can see this in action if you review the log of any of the init containers:
Select any image to see a larger version.
Mobile users: To view the images, select the "Full" version at the bottom of the page.
The current enforced startup order is summarized below. The software essential proceeds through each stage of the process until all services are running.
NOTE: the diagram does not detail every service in the environment.
The start sequencer adds a label to each POD where it is implemented. To see if you have the start-sequencer implemented in your environment you can run the following command, if any pods are returned, then the start sequencer is implemented.
kubectl get pods -l sas.com/start-sequencer=true -n mynamespace
Limited testing shows some improvement in startup time, but for the moment the sample size is very small. I expect we might see more improvement in environments where resources are constrained. If you do implement this feature, please report back with your experiences.
Find more articles from SAS Global Enablement and Learning here.
