A generic method for copying backup packages for migration or disaster recovery

1 Like

SAS Viya 2021.1.4 includes a new generic approach to making a backup package available to a target environment in a SAS Viya migration or disaster recovery. The 20201.1.4 documentation includes instructions on how to use this new approach which should work with only minor differences regardless of the cloud provider of the target SAS Viya environment. In this post, I will look at this new approach

I discussed backup and restore in SAS Viya in a previous article. In that post, I described how a Viya backup is stored on two main persistent volumes claims(PVC) within the k8s cluster (depending on what software you have deployed there could be additional volumes.):

sas-cas-backup-data contains the backup package for the CAS server including the default caslibs and the CAS permstore
sas-common-backup-data contains the backup of the Viya Configuration Server and Infrastructure Data Server

You can use kubectl to see the volumes in your deployment.

kubectl get pvc -l "sas.com/backup-role=storage" -n $current-namespace

Select any image to see a larger version.
Mobile users: To view the images, select the "Full" version at the bottom of the page.

Each individual backup within these PVC's is in a directory which is named with a DateTime stamp representing the backup id of the backup. It is these directories that represent the backup package and that must be copied to the target deployment for a restore.

How does this new approach work?

To copy the backup package to the target Kubernetes cluster the kubectl cp command is used. Kubectl cp is a command that can copy files and directories to and from containers (similar to SCP in Linux). The basic syntax (below) specifies a source and target destination for the copy operation which refer to either a local or remote destination.

kubectl cp   [file-spec-src] [file-spec-dest]

The format to specify the remote destination is: <k8s-pod-name>:<file-or-dir>. As an example, to copy a single file (SAS dataset) into the CAS public directory mounted into the CAS controller you can run the following kubectl cp command.

kubectl -n {TARGETNS} cp /mnt/workshop_files/dataforpublic/hmeq_test.sas7bdat sas-cas-server-default-controller:/cas/data/caslibs/public/hmeq_test.sas7bdat

To perform the copy the file or directory must be accessible to the machine where kubectl is running (either directly on the machine or via some kind of share). You also have to be aware of permissions, both inside the pod and of the user identity you use to run the kubectl cp. Make sure:

that the directory and file permissions are the same in the target environment as in the source environment.
the user that issues the kubectl cp command to copy files has appropriate permissions to copy data from the backup package and to the target PVC.

To copy and restore a backup you need to know the backup id. The backup id is stored in a status.json file inside the backup. To identify it follow the instructions in the documentation.

Copy to the CAS backup to sas-cas-backup-data

To copy the CAS backup we can follow this same approach that was used to copy the SAS dataset to the public caslib. The CAS controller pod is always running in a Viya deployment and the containers in the pod have a PVC sas-cas-backup-data This is the location where CAS backups are stored, and also the location where we can copy CAS backups if we wish to restore from them.

To copy the CAS backup to the target deployment it must be accessible to the machine where you are running kubectl.

Step 1) Get the name of the CAS controller pod.

There are a couple of ways to do this. You can check where the sas-cas-backup-data PVC is mounted.

CONTROLLERPOD=$( kubectl -n ${TARGETNS} describe pvc sas-cas-backup-data | grep >"Mounted By" | awk '{print $3}' )
echo ${CONTROLLERPOD}
sas-cas-server-default-controller

OR you can get the name of the controller using this command. The environment variable contains the name of the CAS controller pod in your deployment.

CONTROLLERPOD= $(kubectl get pod -l casoperator.sas.com/node-type=controller --output=jsonpath={.items..metadata.name})
echo ${CONTROLLERPOD}
sas-cas-server-default-controller

Step 2) Use kubectl cp to copy the CAS backup to the target environment.

Use kubectl cp to copy the backup we want to the target. Use the

backup id of the backup
sas-backup-agent container
backup mount point in the container which is /sasviyabackup.

kubectl -n ${TARGETNS} cp /tmp/backup/cas/${backupid} ${CONTROLLERPOD}:/sasviyabackup -c sas-backup-agent

Step 3) Test that the backup package was copied. Test that the backup data has been copied. To do this exec into the pod and run an ls command to display the contents of the /sasviyabackup directory.

kubectl -n ${TARGETNS} exec -it ${CONTROLLERPOD} -c  sas-backup-agent -- bash -c "ls -al /sasviyabackup"

total 0
drwxrwxrwx 3 root root 42 Aug 24 20:58 .
rwxr-xr-x 1 root root 76 Aug 24 20:07 ..
drwxr-xr-x 3 1002 sas  25 Aug 24 20:58 2021-08-24T19_15_27_627_0700

Copy the Common Backup to sas-common-backup-data

The common-backup-data is more complicated and this is where the new functionality comes in. By default, the only time that the sas-common-backup-data PVC is mounted is when a backup or a restore Kubernetes job is running. As a result, there is generally no running POD that can be used as a destination to copy the common backup data.The new functionality provided starts a Kubernetes job that contains a POD that uses sas-common-backup-data PVC. Running the job provides a POD with a target destination for a copy the common backup data.

In 20201.1.4 the overlay defining the job is in the project directory at /sas-bases/examples/backup/sas-backup-copy-job.yaml. Instructions for using it are provided in the readme.md file in the same directory and here in the documentation.

To copy your common backup data to the target environment follow these steps. Step 1) Keep a copy of the current kustomization.yaml to restore after the copy is done and, if it doesn't already exist, create a folder under site-config called backup.

cp kustomization.yaml kustomization.yaml.save
mkdir -p ~/project/deploy/${TARGETNS}/site-config/backup

Step 2) Copy the sas-backup-copy-job.yaml file from /sas-bases/examples to the new ../site-config/backup sub-directory..

cp ~/project/deploy/${TARGETNS}/sas-bases/examples/backup/sas-backup-copy-job.yaml ~/project/deploy/${TARGETNS}/site-config/backup

Step 3). In the backup directory create a kustomization.yaml that includes a reference to the sas-backup-copy-job.yaml overlay. The file should look like this.

Step 4). Update the kustomization.yaml file in the project directory to reference the kustomizations in the backup directory by adding the line -site-config/backup to the resources section.

Step 5) Build the manifest that will start the job.

cd ~/project/deploy/${TARGETNS}
kustomize build -o site-sas-backup-copy-job.yaml

Step 6) Start the job by applying the manifest. The job will start and a POD inside the job will mount the sas-common-backup-data PVC.

kubectl -n ${TARGETNS} apply -f site-sas-backup-copy-job.yaml -l sas.com/job-type=sas-backup-copy-job

job.batch/sas-backup-copy-job created

Step 7) Get the name of the POD that the job creates so that we can use it in the kubectl cp command.

PODNAME=$( kubectl -n ${TARGETNS} get pods -l sas.com/job-type=sas-backup-copy-job -o jsonpath="{.items[0].metadata.name}" )
echo ${PODNAME}
sas-backup-copy-job-8wj9p

Step 8 ) Use kubectl cp to copy the data to the target common data PVC.

kubectl -n ${TARGETNS} cp /tmp/backup/common/${backupid} ${PODNAME}:/sasviyabackup

Step 9). Test that the backup data has been copied.

kubectl -n ${TARGETNS} exec -it ${PODNAME} -- bash -c "ls -al /sasviyabackup"

You should see a directory with the name of the backup id that was copied in.

   total 0
   drwxrwxrwx 3 root root 42 Aug 24 20:58 .
   drwxr-xr-x 1 root root 76 Aug 24 20:07 ..
   drwxr-xr-x 3 1002 sas  25 Aug 24 20:58 2021-08-24T19_15_27_627_0700

Step 10) Lastly restore the original kustomization.yaml and delete the backup copy job.

rm site-sas-backup-copy-job.yaml
mv kustomization.yaml.save kustomization.yaml
kubectl -n ${TARGETNS} delete job -l sas.com/job-type=sas-backup-copy-job

With the package now available in the target PVC you can now perform a restore. This new functionality will help with copying backup packages between Viya environments. For more information check out these resources.

Find more articles from SAS Global Enablement and Learning here.