BookmarkSubscribeRSS Feed

Scheduling Start and Stop Jobs for SAS Viya

Started ‎08-03-2021 by
Modified ‎08-03-2021 by
Views 9,101

Administrators of any software that runs in the cloud will undoubtedly be asked to help reduce costs by shutting down software when it is not in use.  If no one in the company works over the weekend then why pay for cloud resources to keep idle software running? SAS Viya administrators are no different but we have the added burden of needing to gracefully stop SAS Viya processes before shutting down cloud resources.  Starting with SAS Viya 2021.1.1 (May), SAS now provides jobs that make stopping and starting a SAS Viya deployment much easier.  The jobs serve double duty as they can be used to stop and start SAS Viya on a regular schedule or they can be used to create ad hoc stop and start jobs that make managing SAS Viya a breeze.

 

Scheduling Stop and Start Jobs

Let's suppose you are the administrator of a SAS Viya deployment that you know is not used after business hours or on Saturday or Sunday.  You have been asked to bring down SAS Viya resources at the end of each workday and to restart everything each workday morning so it will be ready for people to use when they arrive.

Assuming you have a fresh SAS Viya 2021.1.1 or later deployment, let's take a look at the two cron jobs that are provided for you to manage your deployment.

 

$ kubectl get cronjobs
NAME                       SCHEDULE       SUSPEND   ACTIVE   LAST SCHEDULE   AGE
sas-backup-purge-job       15 0 1/1 * ?   False     0        17h             5d23h
sas-scheduled-backup-job   0 1 * * 0      False     0        41h             5d23h
sas-start-all              0 5 31 2 *     True      0                        5d23h
sas-stop-all               0 5 31 2 *     True      0                        5d23h
sas-update-checker         46 10 * * 4    False     0        4d7h            5d23h

 

Obviously, the two jobs we want to focus on are sas-stop-all and sas-start-all. These jobs are namespace specific so you will have a set of them for each SAS Viya namespace in your Kubernetes cluster.

 

Both jobs are initially created with two safety features that ensure the jobs will not run without an administrator taking proactive steps to activate them.

  • The initial cron specification is set to run at 5:00am on February 31st.
  • The jobs are suspended.

 

So, to carry out your orders, all you need to do is to modify the cron specifications for the two jobs and to unsuspend them. 

 

Step 1: Create site-config/schedule-start-stop.yaml

The first thing you need to do is to create the yaml file to specify the appropriate cron schedules and unsuspend the sas-stop-all and sas-start-all jobs.  (If you are not familiar with cron scheduling syntax you may want to visit crontab.guru for more information.)  As with all site-specific customizations, you should save your yaml file in the site-config directory of your deployment.

 

If you are running SAS Viya 2021.1.1 or 2021.1.2, you will need to craft your own yaml overlay in the site-config directory of your deployment directory and then apply the change.  In this example, I named the file site-config/schedule-start-stop.yaml.

 

If you are running SAS Viya 2021.1.3 or later, there is an example schedule-start-stop.yaml overlay in sas-bases/examples/kubernetes-tools/lifecycle-operations/schedule-start-stop you can copy and modify.

 

cd ~/project/deploy/gelcorp
cp sas-bases/examples/kubernetes-tools/lifecycle-operations/schedule-start-stop/schedule-start-stop.yaml ./site-config
chmod ug+w ./site-config/schedule-start-stop.yaml

 

Regardless of the method you use, the contents of site-config/schedule-start-stop.yaml should look like this.

 

---
apiVersion: builtin
kind: PatchTransformer
metadata:
  name: schedule-stop-all
patch: |-
  - op: replace
    path: /spec/schedule
    # schedule: MUST BE PROVIDED BY USER.
    # This is the cron schedule by which the sas-stop-all job is run.
    # Example:
    #   value: '0 19 * * 1-5'
    value: '0 23 * * 1-5'
  - op: replace
    path: /spec/suspend
    value: false
  - op: replace
    path: /metadata/labels/sas.com~1deployment
    value: 'user-specified'
target:
  name: sas-stop-all
  kind: CronJob
---
apiVersion: builtin
kind: PatchTransformer
metadata:
  name: schedule-start-all
patch: |-
  - op: replace
    path: /spec/schedule
    # schedule: MUST BE PROVIDED BY USER.
    # This is the cron schedule by which the sas-start-all job is run
    # Example:
    #   value: '0 7 * * 1-5'
    value: '0 11 * * 1-5'
  - op: replace
    path: /spec/suspend
    value: false
  - op: replace
    path: /metadata/labels/sas.com~1deployment
    value: 'user-specified'
target:
  name: sas-start-all
kind: CronJob

 

One note about the cron schedule value. The default time zone for pods is UTC so you will need to take that into account when you set the job values. In this example, I am in Eastern Daylight Time which is UTC-4 so the values in my yaml file are set to stop SAS Viya at 1900 local time and restart it at 0700 local time.

 

If you want to verify the default pod time zone you can exec into any of the SAS Viya pods and run the date command to print the time zone from inside the pod.

$ kubectl exec sas-scheduler-67d76c58d-4snsw -it -- date
Thu Jul 15 18:37:51 UTC 2021

 

Step 2: Add your patch transform to the kustomization.yaml

The next step is to add your site-config/schedule-start-stop.yaml file to the transformers section of kustomization.yaml.

 

Important note: The reference to site-config/schedule-start-stop.yaml must follow sas-bases/overlays/required/transformers.yaml.

 

$ cat kustomization.yaml
namespace: gelcorp
resources:
  - sas-bases/base
  - site-config/security/cert-manager-issuer
  - sas-bases/overlays/network/ingress
  - sas-bases/overlays/network/ingress/security
  - sas-bases/overlays/internal-postgres
  - sas-bases/overlays/crunchydata
  - sas-bases/overlays/cas-server
  - sas-bases/overlays/internal-elasticsearch 
  - sas-bases/overlays/update-checker # added update checker
  - site-config/sas-prepull/add-prepull-cr-crb.yaml
configurations:
  - sas-bases/overlays/required/kustomizeconfig.yaml
transformers:
  - sas-bases/overlays/network/ingress/security/transformers/product-tls-transformers.yaml 
  - sas-bases/overlays/network/ingress/security/transformers/ingress-tls-transformers.yaml 
  - sas-bases/overlays/network/ingress/security/transformers/backend-tls-transformers.yaml 
  - sas-bases/overlays/internal-elasticsearch/sysctl-transformer.yaml
  - sas-bases/overlays/required/transformers.yaml
  - sas-bases/overlays/internal-postgres/internal-postgres-transformer.yaml
  - site-config/security/cert-manager-provided-ingress-certificate.yaml 
  - site-config/schedule-start-stop.yaml  # Schedule stop and start jobs
  ...

 

Step 3: Build and apply the change

As with any change to kustomization.yaml, you now need to build the deployment manifest file and apply it to your environment.  

 

$ kustomize build -o site.yaml
$ kubectl apply -f site-yaml

 

Step 4: Validate the new job definitions

After applying your new patch, you should be able to validate that the sas-stop-all and sas-start-all jobs are scheduled to run at the requested times and that both jobs are no longer suspended.  

 

$ kubectl get cronjobs
NAME                       SCHEDULE       SUSPEND   ACTIVE   LAST SCHEDULE   AGE
sas-backup-purge-job       15 0 1/1 * ?   False     0        18h             5d23h
sas-scheduled-backup-job   0 1 * * 0      False     0        41h             5d23h
sas-start-all              0 11 * * 1-5   False     0                        5d23h
sas-stop-all               0 23 * * 1-5   False     0                        5d23h
sas-update-checker         46 10 * * 4    False     0        4d7h            5d23h

 

That's it.  Of course you will probably want to notify users of the regularly scheduled hours of operation but at this point, your SAS Viya deployment will stop gracefully each weekday at 23:00pm UTC and restart at 11:00am UTC each weekday.

 

But wait!  There's more...

 

Ad-hoc stopping and starting of SAS Viya

For my money, the best part about having these two convenience jobs is the ability to leverage them to immediately stop or start my SAS Viya deployment.  No more fiddling with repeated kustomization.yaml changes to gracefully stop SAS Viya.  Now you can simply create a new job definition modeled on either sas-stop-all or sas-start-all and you can easily manage your deployment with a single command.

 

For example, let's say that you have the need to immediately stop your SAS Viya deployment.  All you need to do is to create a new job from the sas-stop-all cron job definition and your job is done. You can name the job anything you like as long as it is unique. In this example, I have created the job with a name that includes the epoch to ensure the job name is unique and to allow me to distinguish between repeated stops.  

 

$ kubectl create job sas-stop-now-`date +%s` --from cronjob/sas-stop-all
job.batch/sas-stop-now-1623767798 created

 

Be aware that this technique immediately runs the new job so do not issue this command until you are ready for your system to stop. Similarly, when you are ready to restart your deployment you can use the same technique to create a start job based on sas-start-all.  

 

$ kubectl create job sas-start-now-`date +%s` --from cronjob/sas-start-all
job.batch/sas-start-now-1623768281 created

 

Key Points...

  • SAS Viya administrators now have an easy way to schedule stopping and starting their deployments
  • The same jobs can be leveraged to perform ad-hoc stopping and starting of SAS Viya processes.
  • The jobs are available in the SAS Viya Stable 2021.1.1 (May) release and later and will be available in the SAS Viya LTS 2021.2 release later this year.
  • SAS Viya Stable 2021.1.3 (July) will provide an example schedule-start-stop.yaml overlay you can use as a starting point. Prior to that release, you will need to write your own yaml file based on the example above.

 

Additional Information

The sas-stop-all and sas-start-all jobs are provided by the lifecycle-operations component of SAS Viya.  You can learn more about lifecycle-operations by reading $deploy/sas-bases/examples/kubernetes-tools/README.md  SAS Viya documentation on starting and stopping a deployment can be found here.

 

Find more articles from SAS Global Enablement and Learning here.

Comments

great paper about the famous kubectl for quick launch of Viya 4 admin jobs!

Thanks, this is a better explanation than what is in the doc. I am having issues with the jobs still being suspended, and I think it has to do with using the viya4-deployment github automated deployment. Seems like the automated deployment doesn't put the transform after the sas-bases/overlays/required/transformers.yaml.

Thanks, appreciate you saying that. If the transform is being added incorrectly it is probably worth opening an issue on the viya-deployment project to raise their awareness to the problem. I'm sure this isn't the only transform that needs to follow sas-bases/overlays/required/transformers.yaml so they will want to know about your experience.

Looks like the job creates a warning now with the most recent release(unless something is out of sync on my end):

 

Warning: spec.template.metadata.annotations[seccomp.security.alpha.kubernetes.io/pod]: deprecated since v1.19; use the "seccompProfile" field instead

 

Is this an issue, or is it fine to ignore? Thanks.

 

According to our current Technical Support notes you should be ok to ignore that warning.
Version history
Last update:
‎08-03-2021 10:35 AM
Updated by:
Contributors

SAS Innovate 2025: Call for Content

Are you ready for the spotlight? We're accepting content ideas for SAS Innovate 2025 to be held May 6-9 in Orlando, FL. The call is open until September 25. Read more here about why you should contribute and what is in it for you!

Submit your idea!

Free course: Data Literacy Essentials

Data Literacy is for all, even absolute beginners. Jump on board with this free e-learning  and boost your career prospects.

Get Started