Administrators of any software that runs in the cloud will undoubtedly be asked to help reduce costs by shutting down software when it is not in use. If no one in the company works over the weekend then why pay for cloud resources to keep idle software running? SAS Viya administrators are no different but we have the added burden of needing to gracefully stop SAS Viya processes before shutting down cloud resources. Starting with SAS Viya 2021.1.1 (May), SAS now provides jobs that make stopping and starting a SAS Viya deployment much easier. The jobs serve double duty as they can be used to stop and start SAS Viya on a regular schedule or they can be used to create ad hoc stop and start jobs that make managing SAS Viya a breeze.
Let's suppose you are the administrator of a SAS Viya deployment that you know is not used after business hours or on Saturday or Sunday. You have been asked to bring down SAS Viya resources at the end of each workday and to restart everything each workday morning so it will be ready for people to use when they arrive.
Assuming you have a fresh SAS Viya 2021.1.1 or later deployment, let's take a look at the two cron jobs that are provided for you to manage your deployment.
$ kubectl get cronjobs NAME SCHEDULE SUSPEND ACTIVE LAST SCHEDULE AGE sas-backup-purge-job 15 0 1/1 * ? False 0 17h 5d23h sas-scheduled-backup-job 0 1 * * 0 False 0 41h 5d23h sas-start-all 0 5 31 2 * True 0 5d23h sas-stop-all 0 5 31 2 * True 0 5d23h sas-update-checker 46 10 * * 4 False 0 4d7h 5d23h
Obviously, the two jobs we want to focus on are sas-stop-all and sas-start-all. These jobs are namespace specific so you will have a set of them for each SAS Viya namespace in your Kubernetes cluster.
Both jobs are initially created with two safety features that ensure the jobs will not run without an administrator taking proactive steps to activate them.
So, to carry out your orders, all you need to do is to modify the cron specifications for the two jobs and to unsuspend them.
The first thing you need to do is to create the yaml file to specify the appropriate cron schedules and unsuspend the sas-stop-all and sas-start-all jobs. (If you are not familiar with cron scheduling syntax you may want to visit crontab.guru for more information.) As with all site-specific customizations, you should save your yaml file in the site-config directory of your deployment.
If you are running SAS Viya 2021.1.1 or 2021.1.2, you will need to craft your own yaml overlay in the site-config directory of your deployment directory and then apply the change. In this example, I named the file site-config/schedule-start-stop.yaml.
If you are running SAS Viya 2021.1.3 or later, there is an example schedule-start-stop.yaml overlay in sas-bases/examples/kubernetes-tools/lifecycle-operations/schedule-start-stop you can copy and modify.
cd ~/project/deploy/gelcorp cp sas-bases/examples/kubernetes-tools/lifecycle-operations/schedule-start-stop/schedule-start-stop.yaml ./site-config chmod ug+w ./site-config/schedule-start-stop.yaml
Regardless of the method you use, the contents of site-config/schedule-start-stop.yaml should look like this.
--- apiVersion: builtin kind: PatchTransformer metadata: name: schedule-stop-all patch: |- - op: replace path: /spec/schedule # schedule: MUST BE PROVIDED BY USER. # This is the cron schedule by which the sas-stop-all job is run. # Example: # value: '0 19 * * 1-5' value: '0 23 * * 1-5' - op: replace path: /spec/suspend value: false - op: replace path: /metadata/labels/sas.com~1deployment value: 'user-specified' target: name: sas-stop-all kind: CronJob --- apiVersion: builtin kind: PatchTransformer metadata: name: schedule-start-all patch: |- - op: replace path: /spec/schedule # schedule: MUST BE PROVIDED BY USER. # This is the cron schedule by which the sas-start-all job is run # Example: # value: '0 7 * * 1-5' value: '0 11 * * 1-5' - op: replace path: /spec/suspend value: false - op: replace path: /metadata/labels/sas.com~1deployment value: 'user-specified' target: name: sas-start-all kind: CronJob
One note about the cron schedule value. The default time zone for pods is UTC so you will need to take that into account when you set the job values. In this example, I am in Eastern Daylight Time which is UTC-4 so the values in my yaml file are set to stop SAS Viya at 1900 local time and restart it at 0700 local time.
If you want to verify the default pod time zone you can exec into any of the SAS Viya pods and run the date command to print the time zone from inside the pod.
$ kubectl exec sas-scheduler-67d76c58d-4snsw -it -- date Thu Jul 15 18:37:51 UTC 2021
The next step is to add your site-config/schedule-start-stop.yaml file to the transformers section of kustomization.yaml.
Important note: The reference to site-config/schedule-start-stop.yaml must follow sas-bases/overlays/required/transformers.yaml.
$ cat kustomization.yaml namespace: gelcorp resources: - sas-bases/base - site-config/security/cert-manager-issuer - sas-bases/overlays/network/ingress - sas-bases/overlays/network/ingress/security - sas-bases/overlays/internal-postgres - sas-bases/overlays/crunchydata - sas-bases/overlays/cas-server - sas-bases/overlays/internal-elasticsearch - sas-bases/overlays/update-checker # added update checker - site-config/sas-prepull/add-prepull-cr-crb.yaml configurations: - sas-bases/overlays/required/kustomizeconfig.yaml transformers: - sas-bases/overlays/network/ingress/security/transformers/product-tls-transformers.yaml - sas-bases/overlays/network/ingress/security/transformers/ingress-tls-transformers.yaml - sas-bases/overlays/network/ingress/security/transformers/backend-tls-transformers.yaml - sas-bases/overlays/internal-elasticsearch/sysctl-transformer.yaml - sas-bases/overlays/required/transformers.yaml - sas-bases/overlays/internal-postgres/internal-postgres-transformer.yaml - site-config/security/cert-manager-provided-ingress-certificate.yaml - site-config/schedule-start-stop.yaml # Schedule stop and start jobs ...
As with any change to kustomization.yaml, you now need to build the deployment manifest file and apply it to your environment.
$ kustomize build -o site.yaml $ kubectl apply -f site-yaml
After applying your new patch, you should be able to validate that the sas-stop-all and sas-start-all jobs are scheduled to run at the requested times and that both jobs are no longer suspended.
$ kubectl get cronjobs NAME SCHEDULE SUSPEND ACTIVE LAST SCHEDULE AGE sas-backup-purge-job 15 0 1/1 * ? False 0 18h 5d23h sas-scheduled-backup-job 0 1 * * 0 False 0 41h 5d23h sas-start-all 0 11 * * 1-5 False 0 5d23h sas-stop-all 0 23 * * 1-5 False 0 5d23h sas-update-checker 46 10 * * 4 False 0 4d7h 5d23h
That's it. Of course you will probably want to notify users of the regularly scheduled hours of operation but at this point, your SAS Viya deployment will stop gracefully each weekday at 23:00pm UTC and restart at 11:00am UTC each weekday.
But wait! There's more...
For my money, the best part about having these two convenience jobs is the ability to leverage them to immediately stop or start my SAS Viya deployment. No more fiddling with repeated kustomization.yaml changes to gracefully stop SAS Viya. Now you can simply create a new job definition modeled on either sas-stop-all or sas-start-all and you can easily manage your deployment with a single command.
For example, let's say that you have the need to immediately stop your SAS Viya deployment. All you need to do is to create a new job from the sas-stop-all cron job definition and your job is done. You can name the job anything you like as long as it is unique. In this example, I have created the job with a name that includes the epoch to ensure the job name is unique and to allow me to distinguish between repeated stops.
$ kubectl create job sas-stop-now-`date +%s` --from cronjob/sas-stop-all job.batch/sas-stop-now-1623767798 created
Be aware that this technique immediately runs the new job so do not issue this command until you are ready for your system to stop. Similarly, when you are ready to restart your deployment you can use the same technique to create a start job based on sas-start-all.
$ kubectl create job sas-start-now-`date +%s` --from cronjob/sas-start-all job.batch/sas-start-now-1623768281 created
The sas-stop-all and sas-start-all jobs are provided by the lifecycle-operations component of SAS Viya. You can learn more about lifecycle-operations by reading $deploy/sas-bases/examples/kubernetes-tools/README.md SAS Viya documentation on starting and stopping a deployment can be found here.
Find more articles from SAS Global Enablement and Learning here.
Registration is open! SAS is returning to Vegas for an AI and analytics experience like no other! Whether you're an executive, manager, end user or SAS partner, SAS Innovate is designed for everyone on your team. Register for just $495 by 12/31/2023.
If you are interested in speaking, there is still time to submit a session idea. More details are posted on the website.
Data Literacy is for all, even absolute beginners. Jump on board with this free e-learning and boost your career prospects.