Administrators of any software that runs in the cloud will undoubtedly be asked to help reduce costs by shutting down software when it is not in use. If no one in the company works over the weekend then why pay for cloud resources to keep idle software running? SAS Viya administrators are no different but we have the added burden of needing to gracefully stop SAS Viya processes before shutting down cloud resources. Starting with SAS Viya 2021.1.1 (May), SAS now provides jobs that make stopping and starting a SAS Viya deployment much easier. The jobs serve double duty as they can be used to stop and start SAS Viya on a regular schedule or they can be used to create ad hoc stop and start jobs that make managing SAS Viya a breeze.
Scheduling Stop and Start Jobs
Let's suppose you are the administrator of a SAS Viya deployment that you know is not used after business hours or on Saturday or Sunday. You have been asked to bring down SAS Viya resources at the end of each workday and to restart everything each workday morning so it will be ready for people to use when they arrive.
Assuming you have a fresh SAS Viya 2021.1.1 or later deployment, let's take a look at the two cron jobs that are provided for you to manage your deployment.
$ kubectl get cronjobs
NAME SCHEDULE SUSPEND ACTIVE LAST SCHEDULE AGE
sas-backup-purge-job 15 0 1/1 * ? False 0 17h 5d23h
sas-scheduled-backup-job 0 1 * * 0 False 0 41h 5d23h
sas-start-all 0 5 31 2 * True 0 5d23h
sas-stop-all 0 5 31 2 * True 0 5d23h
sas-update-checker 46 10 * * 4 False 0 4d7h 5d23h
Obviously, the two jobs we want to focus on are sas-stop-all and sas-start-all. These jobs are namespace specific so you will have a set of them for each SAS Viya namespace in your Kubernetes cluster.
Both jobs are initially created with two safety features that ensure the jobs will not run without an administrator taking proactive steps to activate them.
- The initial cron specification is set to run at 5:00am on February 31st.
- The jobs are suspended.
So, to carry out your orders, all you need to do is to modify the cron specifications for the two jobs and to unsuspend them.
Step 1: Create site-config/schedule-start-stop.yaml
The first thing you need to do is to create the yaml file to specify the appropriate cron schedules and unsuspend the sas-stop-all and sas-start-all jobs. (If you are not familiar with cron scheduling syntax you may want to visit crontab.guru for more information.) As with all site-specific customizations, you should save your yaml file in the site-config directory of your deployment.
If you are running SAS Viya 2021.1.1 or 2021.1.2, you will need to craft your own yaml overlay in the site-config directory of your deployment directory and then apply the change. In this example, I named the file site-config/schedule-start-stop.yaml.
If you are running SAS Viya 2021.1.3 or later, there is an example schedule-start-stop.yaml overlay in sas-bases/examples/kubernetes-tools/lifecycle-operations/schedule-start-stop you can copy and modify.
cd ~/project/deploy/gelcorp
cp sas-bases/examples/kubernetes-tools/lifecycle-operations/schedule-start-stop/schedule-start-stop.yaml ./site-config
chmod ug+w ./site-config/schedule-start-stop.yaml
Regardless of the method you use, the contents of site-config/schedule-start-stop.yaml should look like this.
---
apiVersion: builtin
kind: PatchTransformer
metadata:
name: schedule-stop-all
patch: |-
- op: replace
path: /spec/schedule
# schedule: MUST BE PROVIDED BY USER.
# This is the cron schedule by which the sas-stop-all job is run.
# Example:
# value: '0 19 * * 1-5'
value: '0 23 * * 1-5'
- op: replace
path: /spec/suspend
value: false
- op: replace
path: /metadata/labels/sas.com~1deployment
value: 'user-specified'
target:
name: sas-stop-all
kind: CronJob
---
apiVersion: builtin
kind: PatchTransformer
metadata:
name: schedule-start-all
patch: |-
- op: replace
path: /spec/schedule
# schedule: MUST BE PROVIDED BY USER.
# This is the cron schedule by which the sas-start-all job is run
# Example:
# value: '0 7 * * 1-5'
value: '0 11 * * 1-5'
- op: replace
path: /spec/suspend
value: false
- op: replace
path: /metadata/labels/sas.com~1deployment
value: 'user-specified'
target:
name: sas-start-all
kind: CronJob
One note about the cron schedule value. The default time zone for pods is UTC so you will need to take that into account when you set the job values. In this example, I am in Eastern Daylight Time which is UTC-4 so the values in my yaml file are set to stop SAS Viya at 1900 local time and restart it at 0700 local time.
If you want to verify the default pod time zone you can exec into any of the SAS Viya pods and run the date command to print the time zone from inside the pod.
$ kubectl exec sas-scheduler-67d76c58d-4snsw -it -- date
Thu Jul 15 18:37:51 UTC 2021
Step 2: Add your patch transform to the kustomization.yaml
The next step is to add your site-config/schedule-start-stop.yaml file to the transformers section of kustomization.yaml.
Important note: The reference to site-config/schedule-start-stop.yaml must follow sas-bases/overlays/required/transformers.yaml.
$ cat kustomization.yaml
namespace: gelcorp
resources:
- sas-bases/base
- site-config/security/cert-manager-issuer
- sas-bases/overlays/network/ingress
- sas-bases/overlays/network/ingress/security
- sas-bases/overlays/internal-postgres
- sas-bases/overlays/crunchydata
- sas-bases/overlays/cas-server
- sas-bases/overlays/internal-elasticsearch
- sas-bases/overlays/update-checker # added update checker
- site-config/sas-prepull/add-prepull-cr-crb.yaml
configurations:
- sas-bases/overlays/required/kustomizeconfig.yaml
transformers:
- sas-bases/overlays/network/ingress/security/transformers/product-tls-transformers.yaml
- sas-bases/overlays/network/ingress/security/transformers/ingress-tls-transformers.yaml
- sas-bases/overlays/network/ingress/security/transformers/backend-tls-transformers.yaml
- sas-bases/overlays/internal-elasticsearch/sysctl-transformer.yaml
- sas-bases/overlays/required/transformers.yaml
- sas-bases/overlays/internal-postgres/internal-postgres-transformer.yaml
- site-config/security/cert-manager-provided-ingress-certificate.yaml
- site-config/schedule-start-stop.yaml # Schedule stop and start jobs
...
Step 3: Build and apply the change
As with any change to kustomization.yaml, you now need to build the deployment manifest file and apply it to your environment.
$ kustomize build -o site.yaml
$ kubectl apply -f site-yaml
Step 4: Validate the new job definitions
After applying your new patch, you should be able to validate that the sas-stop-all and sas-start-all jobs are scheduled to run at the requested times and that both jobs are no longer suspended.
$ kubectl get cronjobs
NAME SCHEDULE SUSPEND ACTIVE LAST SCHEDULE AGE
sas-backup-purge-job 15 0 1/1 * ? False 0 18h 5d23h
sas-scheduled-backup-job 0 1 * * 0 False 0 41h 5d23h
sas-start-all 0 11 * * 1-5 False 0 5d23h
sas-stop-all 0 23 * * 1-5 False 0 5d23h
sas-update-checker 46 10 * * 4 False 0 4d7h 5d23h
That's it. Of course you will probably want to notify users of the regularly scheduled hours of operation but at this point, your SAS Viya deployment will stop gracefully each weekday at 23:00pm UTC and restart at 11:00am UTC each weekday.
But wait! There's more...
Ad-hoc stopping and starting of SAS Viya
For my money, the best part about having these two convenience jobs is the ability to leverage them to immediately stop or start my SAS Viya deployment. No more fiddling with repeated kustomization.yaml changes to gracefully stop SAS Viya. Now you can simply create a new job definition modeled on either sas-stop-all or sas-start-all and you can easily manage your deployment with a single command.
For example, let's say that you have the need to immediately stop your SAS Viya deployment. All you need to do is to create a new job from the sas-stop-all cron job definition and your job is done. You can name the job anything you like as long as it is unique. In this example, I have created the job with a name that includes the epoch to ensure the job name is unique and to allow me to distinguish between repeated stops.
$ kubectl create job sas-stop-now-`date +%s` --from cronjob/sas-stop-all
job.batch/sas-stop-now-1623767798 created
Be aware that this technique immediately runs the new job so do not issue this command until you are ready for your system to stop. Similarly, when you are ready to restart your deployment you can use the same technique to create a start job based on sas-start-all.
$ kubectl create job sas-start-now-`date +%s` --from cronjob/sas-start-all
job.batch/sas-start-now-1623768281 created
Key Points...
- SAS Viya administrators now have an easy way to schedule stopping and starting their deployments
- The same jobs can be leveraged to perform ad-hoc stopping and starting of SAS Viya processes.
- The jobs are available in the SAS Viya Stable 2021.1.1 (May) release and later and will be available in the SAS Viya LTS 2021.2 release later this year.
- SAS Viya Stable 2021.1.3 (July) will provide an example schedule-start-stop.yaml overlay you can use as a starting point. Prior to that release, you will need to write your own yaml file based on the example above.
Additional Information
The sas-stop-all and sas-start-all jobs are provided by the lifecycle-operations component of SAS Viya. You can learn more about lifecycle-operations by reading $deploy/sas-bases/examples/kubernetes-tools/README.md SAS Viya documentation on starting and stopping a deployment can be found here.
Find more articles from SAS Global Enablement and Learning here.
