You CAN take it with you! Saving SAS Viya Content
I was asked a question recently about how a user would take their content from a running Viya Deployment and save it. The question was raised in the context of SAS Hackathon participants saving the content they created when the event is completed. This is not an uncommon scenario in the world of cloud computing where environments can be spun up and torn down quickly. In this post, I will review the options available for saving and restoring content and provide you with some reference material to help with these tasks.
This is a topic that comes up regularly.
Some other use cases are:
- In GEL in the context of running classes, we need to spin up environments and seed them with the content that the students need to perform the hands-on exercise. SAS Education has this same requirement.
- Providing on-demand SAS environments in the cloud is another good use case.
So how do we deal with this scenario where we work in an environment that has a short lifespan, but we want to make sure we can keep, and restore our content when the environment is retired? There is good news and bad news. The good news is we have a lot of tools available to perform and even automate this process. The bad news is it can be complex and uses a variety of different tools depending on the content and configuration created.
The tools available lead to two main approaches. One saves the whole environment, while the other tries to save only selected content.
Save and Restore everything: Backup and Restore
If you want to save an entire system the SAS Viya Backup and Restore functionality is the best approach. A full-system migration is when you backup and restore to a new environment. This approach is documented in the SAS Help Center: SAS Viya: Full-System Migration
A SAS Viya backup will include most all user-created content and configuration. Including:
- Content saved in the SAS Infrastructure Data Server such as reports (including SAS Visual Analytics reports), comments, authorization rules, attachments, audit records, user preferences, and data source definitions.
- Configuration settings and definitions for infrastructure and applications. These settings are saved in SAS Configuration Server (Consul). Application configuration settings include user interface (UI) themes (for example, the SAS Visual Analytics UI theme) and batch run date (for example, batch date of running SAS Visual Analytics reports).
- CAS access controls, configuration, and global caslib definitions are backed up.
- CAS data stored on the default CAS persistent volumes.
Several items are not covered by the Viya backup. Including:
- Kubernetes manifest and related items.
- User home directories.
- Other data sources, such as data in Third-party databases, Data that is loaded into memory, and Data that is stored on remote file systems (such as path-based data sources including PATH and DNFS).
You can save these items by using other techniques. Data on the file system can be backed up using Kuberbenets and OS (Operating System) commands, in-memory data can be persisted to the file system and backed up, third-party databases are usually governed by their own backup and restore procedures. I covered the details of how-to Backup and Restore in a previous blog post. You can check out this post for a review of the steps involved in backing up and restoring a SAS Viya environment.
A full backup is great, but it is a full backup, you get everything, and you do not get a choice. In addition, the restore process restores the complete environment, again you cannot do a partial restore. If you happen to be the only user of the deployment, as is the case in the SAS Viya Hackathon, this approach may be applicable for you. If you share a deployment with others, then this approach may be less attractive. Fortunately, we have other tools which can choose what to save and load subsets of content.
Select what to Save and Restore: Content Migration
The Content Migration tools support the export and import of Viya Content and Configuration. Using these tools, you can save the important parts of your environment and restore them later. Let us review what can be accomplished for selected content and configuration. Content Migration is documented at the SAS Help Center: SAS Viya: Content Migration from SAS Viya 4
Content in Folders
Most Viya content such as reports, SAS programs, jobs, analytical models, etc. are stored in the SAS Infrastructure Data Server and surfaced in the Viya folder tree.
In most cases, you can choose where in the folders to store your content. Being organized and having a well-defined folder structure and a good content naming convention is a great best practice and can help you when you want to save and restore content. Be aware that there are some types of content that are stored in specific locations. For example, Model Manager saves models in the users “My Folder” and some Visual Analytic content such as themes and data views are saved in the /Products/SAS Visual Analytics area of the folder tree.
Content stored in folders can be easily exported and imported using SAS Environment Manager and/or the transfer plug-in of the sas-viya command-line interface (CLI) The tools create SAS Viya packages. The capability to import and export content was previously restricted to Viya Administrators but was recently opened to all users. This change opens the tools up to allow users to export and import their own content. For details check out this blog post: Non Administrator Access to Content Migration in SAS Viya 4
Exporting
In the screenshot below is a folder containing sub-folders with a variety of content (flows, job, queries, reports, data plans, etc.)
Select any image to see a larger version.
Mobile users: To view the images, select the "Full" version at the bottom of the page.
To export a folder and all of its contents to a SAS Viya package, open the Content page of SAS Environment Manager, select the folder, then right-click and select Export.
A SAS Viya Package file is saved in the Browsers download directory.
The sas-viya CLI can be used to export content from the command line. In this case, it is a multi-step process. The steps shown below in the code are:
1. Identify the URI (a unique identifier) of the folder
2. Export the folder using its URI
3. Identity the packageid of the package created. This step is shown programmatically below but the id is also output to the terminal window when you run the export step.
4. Download from the Infrastructure Data Server the package created by the export.
folderid=$(sas-viya --output json folders show --path /gelcontent/GELCorp/Sales | jq | jq -r '.["id"]')
sas-viya --output text transfer export -u /folders/folders/$folderid --name SalesContent$folderid
packageid=$(sas-viya --output json transfer list --name SalesContent$folderid | jq | jq -r '.items[]["id"]')
sas-viya transfer download -f /tmp/viya4packages/SalesContent.json --id $packageid
The GEL pyviyatools are a set of command-line tools that call the SAS Viya REST API from python. The tools are available on the SAS GitHub site. The tools can be used to make direct calls to any rest-endpoint (like a CURL command) or to build additional tools that make multiple rest calls to provide more complex functionality. The GEL pyviyatools include several tools that can help with the process of saving and reloading configuration and content.
To export a folder, we can use one command.
exportfolder.py -f "/gelcontent/GELCorp/Sales/Reports" -d /tmp/viya4packages --filename SalesContent.json -q
In the example, we exported a subset of content (a folder and its content). What if you want to export the complete folder structure but you do not want to perform a full backup? The pyviyatools can also help with this. The following command will export all the folders in the folder tree and their sub-folders to a directory.
exportfoldertree.py -d /tmp/allfolders -q
The output directory will contain a Viya transfer package for each sub-folder of the SAS Content root
Importing
The same tools that were used for export can be used on the import side to restore your content. SAS Environment Manager import interface supports importing any SAS transfer package regardless of how it was created. Select Import in the UI and walk through the steps in the Wizard to import a package, during the import you will have the opportunity to remap to the same or different resources in the target environment.
With the sas-viya cli again it is a multi-step process to import a package. The steps:
- Upload the package to the target environments Infrastructure Data Server
- Import the package.
packageid=$(sas-viya --output json transfer upload --file /tmp/viya4packages/SalesContent.json | jq | jq -r '.["id"]')
sas-viya --output text transfer import --id $packageid
Pyviyatools again can simplify the process. Copy any packages you want to import to a directory and the tool will import all packages in the directory. For example, the command below will import all JSON packages stored in the /tmp/mypackages directory.
importpackages.py -d /tmp/mypackages/ -qThe transfer service will deal with any object in Viya that has a URI Uniform Resource Identifiers(URI). There are some objects that do not exist in folders but do have URI’s, for example, authorization rules, custom groups, or job requests. These objects can be exported and imported using the transfer plug-in of the sas-viya cli and imported using the SAS Environment Manager Import interface. For more details you can check out these blog posts:
- Uniform Resource Identifiers(URI) in SAS Viya
- Promotion of Viya resources that are not stored in folders
- Promoting Viya Job Flows
Specific Content Considerations
Certain types of content have additional considerations you should be aware of. These considerations are documented in content-specific migration guides. In addition, you can review these posts from my colleague @BethEbersole :
- Moving SAS Model Studio Custom Pipelines to a New Environment: Not Everything Conveys
- Managing Models in Viya 3.3: A Few Highlights
| Model Studio | Promotions and Upgrades within SAS Viya in SAS Visual Data Mining and Machine Learning: Advanced Topics Promotions and Upgrades within SAS Viya in SAS Visual Text Analytics: User’s Guide |
| SAS Visual Text Analytics | Promotions Considerations Specific to SAS Visual Text Analytics in SAS Visual Text Analytics: User’s Guide |
| SAS Visual Forecasting | Promotions and Upgrades within SAS Viya in Model Studio: SAS® Visual Forecasting Guide |
Caslibs and Data
CAS libraries (caslibs) are:
- used to access data in the CAS server.
- do not reside in folders in Viya.
There are two aspects to caslibs. The:
- caslib metadata that contains the caslib definition, properties, authorization, etc., AND
- actual data itself.
Caslib metadata
The cas plugin of the sas-viya command-line interface can be used to export and import caslib definitions and authorization.
Output the definition of the salesdl caslib to a JSON file.
sas-viya --output json cas caslibs show-info --name "salesdl" --server=cas-shared-default > salescaslib.jsonImport the definition of the salesdl caslib from a JSON file.
sas-viya cas caslibs create path --source-file /tmp/salescaslib.json
In addition (you might have guessed :)), there are a few pyviyatools that can help to automate this process. exportcaslibs: will export all or a subset of caslibs to JSON files in a directory. It can optionally include the caslib authorization settings.
exportcaslibs.py -s cas-shared-default -d /tmp/mycaslibs -dc "gelcontent" -i -q
importcaslibs: pass in a directory the tool will create caslibs from the JSON caslib definitions in the directory. If authorization files exist it will also apply the caslib authorization settings.
Physical Data
Regarding physical data, a lot will depend on how the data is stored. Viya on Kubernetes makes it more complicated as the data may reside on persistent volumes (PV) in the cluster and not be as readily accessible as it was in previous versions of the software. For a good explanation of how persistent volumes work with Viya check out this post. There are options from within the SAS Viya Software, proc CAS, and proc CASTUTIL allow you to save and copy data as do proc DATASETS and proc COPY in base SAS. Data can also be downloaded from SAS Environment Manager. Obviously, there is a practical limit to how much data can be downloaded from the browser. If your data is not too big then go to the Data pane of Environment Manager select the in-memory table, right-click and select Download table.
Depending on the permissions you have in Kubernetes you can use kubectl cp to copy data from the cluster to a local drive and vice-versa. Kubectl cp is a command that can copy files and directories to and from containers (like SCP in Linux). The basic syntax (below) specifies a source and target destination for the copy operation which refers to either a local or remote destination. The code below copies a SAS dataset from the Public CASLIB to a directory on the local drive.
kubectl -n myns cp sas-cas-server-default-controller:/cas/data/caslibs/public/hmeq.sas7bdat -c cas /tmp/downloads/hmeq.sas7bdatA couple of blog posts from my colleague Stephen Foerster focus on getting data into Viya on Kuberenetes, the concepts discussed also apply to getting data out.
It is possible that you have made changes to Viya configuration that you want to save. In that case, you can use the techniques discussed in this post: Saving and reloading Viya Configuration
Conclusion
Focusing on the good news then, it is very possible to save your Viya content (and configuration) and reload it. You have options. I hope in this post I have helped you understand your options and pointed you to some resources that can help you with this process.
Find more articles from SAS Global Enablement and Learning here.
