A first look and Backup and Restore in SAS Viya on Kubernetes

3 Likes

Backup and restore is always a critical feature of enterprise software. In the new SAS Viya (2020.1 and later) the architecture and implementation of this feature has changed significantly from previous releases. In this "first look" post I will provide an overview of backup and restore in the latest release of SAS Viya.

In SAS Viya 3.x backup and restore was available from SAS Environment Manager and from the backup plug-in of the sas-admin CLI. This is not the case in SAS Viya 4. In the latest release of SAS Viya backup and restore are implemented using native Kubernetes functionality. Backup and restore are implemented as Kubernetes jobs. Kuberenetes jobs are used when you want to create pods to perform a specific task to completion and then exit. Backup and restore is a common use case for jobs.

So how do we create and run these jobs? As with many administration tasks, we build and then apply a Kuberenetes manifest to make backup and restore happen. If I just lost you please check out this article for an overview of building manifests with Kuztomize.

Before we look at the process, let's look at what is backed up, what is not backed up, when it is backed up, and where it goes.

A SAS Viya backup includes:

Content stored in the Infrastructure Data Server (reports, folders,rules, etc.)
Configuration stored in the SAS Configuration Server (application configuration settings etc.)
CAS permstore (Caslib definitions, access controls, etc.)
Default CAS persistent volume content (contents of the Public caslib etc.)

Not included in the backup are the contents of your project directory (Kubernetes manifests), user home directories, other data sources such as the source of path-based caslibs, third-party databases, etc.

The default backup is a cronjob (a kubernetes job that runs on a schedule), it runs at 1 am every Sunday but you can change this schedule. Backups are written to and read from Kubernetes persistent volumes. The deployment process creates two persistent volumes:

sas-cas-backup-data
sas-common-backup-data

You can use kubectl to see the volumes in your deployment.

kubectl get pvc -l "sas.com/backup-role=storage" -n $current-namespace

Select any image to see a larger version.
Mobile users: To view the images, select the "Full" version at the bottom of the page.

It's all a bit abstract at this point so let's see how we would perform an ad-hoc backup. An ad-hoc backup can be run at any time, potentially as a backup before making a change to the system, or as the start of a migration to a new system.

Perform an ad-hoc backup

To perform an ad-hoc backup job you can start the scheduled backup on an ad-hoc basis To do this use kubectl to run the following command. The command will execute the regularly scheduled backup immediately.

kubectl create job --from=cronjob/sas-scheduled-backup-job sas-scheduled-backup-job-adhoc-001 -n $current-namespace

To get the status of the backup

I

kubectl get jobs -l "sas.com/backup-job-type=scheduled-backup" -L "sas.com/backup-job-type,sas.com/sas-backup-job-status" -n $current-namespace

The backup package will be on the two persistent volume claims mentioned earlier

The sas-common-backup-data-pvc contains a sub-directory named for the unique backup id of this backup. Navigating through the sub-directories you will find a status.json file that summarizes the status of the backup and two directories:

consul contains the backup of the Configuration Server
postgres contains the backup of the Infrastructure Data Server

The sas-cas-backup-data-pvc has a sub-directory named for the unique id of this backup (backupid). Within the sub-directories are two main directories.

cas contains a sub-directory for each CAS server, that sub-directory contains the CAS permstore for that server. The permstore contains the casllb defintions, authorization etc.
filesystem contains a sub-directory for each CAS server, within the sub-directory is a caslibs directory that contains the data from the SAS Viya default caslibs, e.g. public, samples, sysData, vamodels, etc.

gn_viya4_backup03_04(1).png

When the backup is complete you will want to remove the reference to the backup job from kustomization.yaml. If you don't do this it will be included in the next manifest you generate and run when that manifest is applied.

Perform a Restore

Restore is similar to backup but it is a two-stage process. If you are restoring to a different system you need to have a running SAS Viya deployment to restore to, and you need to make the backup package available in the persistent volume on the target system. During the restore process the environment will be (mostly) be stopped. To restore the process requires you to create two Kuberenetes manifests that are applied in separate steps. It can be a little confusing as you create the manifests in the opposite order from which you apply them.

Create restore.yaml

Restarts the services and starts the CAS server in restore mode. Edit ~/project/deploy{$current_namespace}/kustomization.yaml to define a few values in a configMapGenerator to:

SAS_BACKUP_ID to identify the backup package using the backup id
SAS_LOG_LEVEL to set the log level(optional)
SAS_DEPLOYMENT_START_MODE to trigger the restore.

The following should be added to the kuztomization.yaml file.

configMapGenerator:
  [... previous resource items ...]
  - name: sas-restore-job-parameters
    behavior: merge
    literals:
      - SAS_BACKUP_ID=${backupid}
      - SAS_DEPLOYMENT_START_MODE=RESTORE
      - SAS_LOG_LEVEL=DEBUG

Now run the build to create the site-restore.yaml manifest

cd ~/project/deploy/${current_namespace}
kustomize build -o site-restore.yaml

Create restore-initial.yaml

Stops all services except SAS Configuration Server (Consul) and SAS Infrastructure Data Server (PostgreSQL) and starts the restore job To create restore-initial edit the kustomization.yaml adding two overlays. In the resources section add the restore overlays.

resources:
   - sas-bases/overlays/restore

n the transformers section add the zero replicas transform that will scale most services (except the Configuration Server and the Infrastructure Data Server to zero).

transformers:
   - sas-bases/overlays/restore/restore-replicas-zero-transformer.yaml

Now run the build to create the site-restore-initial.yaml manifest

cd ~/project/deploy/${current_namespace}
kustomize build -o site-restore-initial.yaml

Run the Restore Process

To run the restore you apply the manifests in order Step 1 apply site-restore-initial.yaml stops most services and start the restore job which restores the Infrastructure Data Server and the Configuration server. You can check on the restore job using kubectl, the status will change to completed when it is done.

kubectl get jobs -l "sas.com/backup-job-type=restore" -L "sas.com/sas-backup-id,sas.com/backup-job-type,sas.com/sas-restore-status" -n ${current_namespace}

backup status

View the logs of the restore job.

kubectl logs -f -l "job-name=sas-restore-job" -c sas-restore-job -n ${current_namespace}

Step 2 apply site-restore.yaml. This manifest will start the:

Environment services by scaling up all resources
CAS Server in restore mode which will restore the data and configuration from the backup package.

In order to restore the CAS file system, the restore requires a clean CAS data volume. List the PVC for reference and then delete the persistent volume claims for CAS

kubectl get pvc -l 'sas.com/backup-role=provider' -n ${current_namespace}
kubectl delete pvc -l 'sas.com/backup-role=provider' -n ${current_namespace}

Check the log to see if the CAS server performed the restore. The logs should show the start of the restore process that restores the backup content to the target CAS persistent volumes.

kubectl logs sas-cas-server-cas-shared-default-controller -c cas -n ${current_namespace} | grep -A 10 "RESTORE"

Check and see if the permstore was restored. Messages should indicate that the CAS permstore is restored from the backup package.

kubectl logs sas-cas-server-cas-shared-default-controller -c cas -n ${current_namespace} | grep -B 1 -A 5 "restoring permstore"

Uses of backup and restore in SAS Viya

In addition to scheduled and ad-hoc backups the backup and restore tools are also the way that SAS Viya supports migration between two SAS Viya environments and disaster recovery. I hope you have found this post a useful introduction. Here are some additional resources:

Find more articles from SAS Global Enablement and Learning here.