Administrators of any software that runs in the cloud will undoubtedly be asked to help reduce costs by shutting down software when it is not in use. If no one in the company works over the weekend then why pay for cloud resources to keep idle software running? SAS Viya administrators are no different but we have the added burden of needing to gracefully stop SAS Viya processes before shutting down cloud resources. Starting with SAS Viya 2021.1.1 (May), SAS now provides jobs that make stopping and starting a SAS Viya deployment much easier. The jobs serve double duty as they can be used to stop and start SAS Viya on a regular schedule or they can be used to create ad hoc stop and start jobs that make managing SAS Viya a breeze.
Let's suppose you are the administrator of a SAS Viya deployment that you know is not used after business hours or on Saturday or Sunday. You have been asked to bring down SAS Viya resources at the end of each workday and to restart everything each workday morning so it will be ready for people to use when they arrive.
Assuming you have a fresh SAS Viya 2021.1.1 or later deployment, let's take a look at the two cron jobs that are provided for you to manage your deployment.
$ kubectl get cronjobs
NAME SCHEDULE SUSPEND ACTIVE LAST SCHEDULE AGE
sas-backup-purge-job 15 0 1/1 * ? False 0 17h 5d23h
sas-scheduled-backup-job 0 1 * * 0 False 0 41h 5d23h
sas-start-all 0 5 31 2 * True 0 5d23h
sas-stop-all 0 5 31 2 * True 0 5d23h
sas-update-checker 46 10 * * 4 False 0 4d7h 5d23h
Obviously, the two jobs we want to focus on are sas-stop-all and sas-start-all. These jobs are namespace specific so you will have a set of them for each SAS Viya namespace in your Kubernetes cluster.
Both jobs are initially created with two safety features that ensure the jobs will not run without an administrator taking proactive steps to activate them.
So, to carry out your orders, all you need to do is to modify the cron specifications for the two jobs and to unsuspend them.
The first thing you need to do is to create the yaml file to specify the appropriate cron schedules and unsuspend the sas-stop-all and sas-start-all jobs. (If you are not familiar with cron scheduling syntax you may want to visit crontab.guru for more information.) As with all site-specific customizations, you should save your yaml file in the site-config directory of your deployment.
If you are running SAS Viya 2021.1.1 or 2021.1.2, you will need to craft your own yaml overlay in the site-config directory of your deployment directory and then apply the change. In this example, I named the file site-config/schedule-start-stop.yaml.
If you are running SAS Viya 2021.1.3 or later, there is an example schedule-start-stop.yaml overlay in sas-bases/examples/kubernetes-tools/lifecycle-operations/schedule-start-stop you can copy and modify.
cd ~/project/deploy/gelcorp
cp sas-bases/examples/kubernetes-tools/lifecycle-operations/schedule-start-stop/schedule-start-stop.yaml ./site-config
chmod ug+w ./site-config/schedule-start-stop.yaml
Regardless of the method you use, the contents of site-config/schedule-start-stop.yaml should look like this.
---
apiVersion: builtin
kind: PatchTransformer
metadata:
name: schedule-stop-all
patch: |-
- op: replace
path: /spec/schedule
# schedule: MUST BE PROVIDED BY USER.
# This is the cron schedule by which the sas-stop-all job is run.
# Example:
# value: '0 19 * * 1-5'
value: '0 23 * * 1-5'
- op: replace
path: /spec/suspend
value: false
- op: replace
path: /metadata/labels/sas.com~1deployment
value: 'user-specified'
target:
name: sas-stop-all
kind: CronJob
---
apiVersion: builtin
kind: PatchTransformer
metadata:
name: schedule-start-all
patch: |-
- op: replace
path: /spec/schedule
# schedule: MUST BE PROVIDED BY USER.
# This is the cron schedule by which the sas-start-all job is run
# Example:
# value: '0 7 * * 1-5'
value: '0 11 * * 1-5'
- op: replace
path: /spec/suspend
value: false
- op: replace
path: /metadata/labels/sas.com~1deployment
value: 'user-specified'
target:
name: sas-start-all
kind: CronJob
One note about the cron schedule value. The default time zone for pods is UTC so you will need to take that into account when you set the job values. In this example, I am in Eastern Daylight Time which is UTC-4 so the values in my yaml file are set to stop SAS Viya at 1900 local time and restart it at 0700 local time.
If you want to verify the default pod time zone you can exec into any of the SAS Viya pods and run the date command to print the time zone from inside the pod.
$ kubectl exec sas-scheduler-67d76c58d-4snsw -it -- date
Thu Jul 15 18:37:51 UTC 2021
The next step is to add your site-config/schedule-start-stop.yaml file to the transformers section of kustomization.yaml.
Important note: The reference to site-config/schedule-start-stop.yaml must follow sas-bases/overlays/required/transformers.yaml.
$ cat kustomization.yaml
namespace: gelcorp
resources:
- sas-bases/base
- site-config/security/cert-manager-issuer
- sas-bases/overlays/network/ingress
- sas-bases/overlays/network/ingress/security
- sas-bases/overlays/internal-postgres
- sas-bases/overlays/crunchydata
- sas-bases/overlays/cas-server
- sas-bases/overlays/internal-elasticsearch
- sas-bases/overlays/update-checker # added update checker
- site-config/sas-prepull/add-prepull-cr-crb.yaml
configurations:
- sas-bases/overlays/required/kustomizeconfig.yaml
transformers:
- sas-bases/overlays/network/ingress/security/transformers/product-tls-transformers.yaml
- sas-bases/overlays/network/ingress/security/transformers/ingress-tls-transformers.yaml
- sas-bases/overlays/network/ingress/security/transformers/backend-tls-transformers.yaml
- sas-bases/overlays/internal-elasticsearch/sysctl-transformer.yaml
- sas-bases/overlays/required/transformers.yaml
- sas-bases/overlays/internal-postgres/internal-postgres-transformer.yaml
- site-config/security/cert-manager-provided-ingress-certificate.yaml
- site-config/schedule-start-stop.yaml # Schedule stop and start jobs
...
As with any change to kustomization.yaml, you now need to build the deployment manifest file and apply it to your environment.
$ kustomize build -o site.yaml
$ kubectl apply -f site-yaml
After applying your new patch, you should be able to validate that the sas-stop-all and sas-start-all jobs are scheduled to run at the requested times and that both jobs are no longer suspended.
$ kubectl get cronjobs
NAME SCHEDULE SUSPEND ACTIVE LAST SCHEDULE AGE
sas-backup-purge-job 15 0 1/1 * ? False 0 18h 5d23h
sas-scheduled-backup-job 0 1 * * 0 False 0 41h 5d23h
sas-start-all 0 11 * * 1-5 False 0 5d23h
sas-stop-all 0 23 * * 1-5 False 0 5d23h
sas-update-checker 46 10 * * 4 False 0 4d7h 5d23h
That's it. Of course you will probably want to notify users of the regularly scheduled hours of operation but at this point, your SAS Viya deployment will stop gracefully each weekday at 23:00pm UTC and restart at 11:00am UTC each weekday.
But wait! There's more...
For my money, the best part about having these two convenience jobs is the ability to leverage them to immediately stop or start my SAS Viya deployment. No more fiddling with repeated kustomization.yaml changes to gracefully stop SAS Viya. Now you can simply create a new job definition modeled on either sas-stop-all or sas-start-all and you can easily manage your deployment with a single command.
For example, let's say that you have the need to immediately stop your SAS Viya deployment. All you need to do is to create a new job from the sas-stop-all cron job definition and your job is done. You can name the job anything you like as long as it is unique. In this example, I have created the job with a name that includes the epoch to ensure the job name is unique and to allow me to distinguish between repeated stops.
$ kubectl create job sas-stop-now-`date +%s` --from cronjob/sas-stop-all
job.batch/sas-stop-now-1623767798 created
Be aware that this technique immediately runs the new job so do not issue this command until you are ready for your system to stop. Similarly, when you are ready to restart your deployment you can use the same technique to create a start job based on sas-start-all.
$ kubectl create job sas-start-now-`date +%s` --from cronjob/sas-start-all
job.batch/sas-start-now-1623768281 created
The sas-stop-all and sas-start-all jobs are provided by the lifecycle-operations component of SAS Viya. You can learn more about lifecycle-operations by reading $deploy/sas-bases/examples/kubernetes-tools/README.md SAS Viya documentation on starting and stopping a deployment can be found here.
Find more articles from SAS Global Enablement and Learning here.
great paper about the famous kubectl for quick launch of Viya 4 admin jobs!
Thanks, this is a better explanation than what is in the doc. I am having issues with the jobs still being suspended, and I think it has to do with using the viya4-deployment github automated deployment. Seems like the automated deployment doesn't put the transform after the sas-bases/overlays/required/transformers.yaml.
Looks like the job creates a warning now with the most recent release(unless something is out of sync on my end):
Warning: spec.template.metadata.annotations[seccomp.security.alpha.kubernetes.io/pod]: deprecated since v1.19; use the "seccompProfile" field instead
Is this an issue, or is it fine to ignore? Thanks.
Are you ready for the spotlight? We're accepting content ideas for SAS Innovate 2025 to be held May 6-9 in Orlando, FL. The call is open until September 25. Read more here about why you should contribute and what is in it for you!
Data Literacy is for all, even absolute beginners. Jump on board with this free e-learning and boost your career prospects.