Managing SAS Viya Transfer Packages
In SAS Viya we can move content between environments by creating transfer packages. Transfer packages are created in the SAS Viya Infrastructure Data server and can be downloaded to a transfer package JSON file on the file system for movement between environments. To manage transfer package files in the file system, you can copy, move and delete the packages as you would any file. However, for the transfer packages stored in the Infrastructure Data server management is a little more complex. In this post, I will look at managing the transfer packages that are created and saved in the Infrastructure Data Server during the import and export of Viya Content.
Export and import of content in Viya can be performed using SAS Environment Manager or the transfer plugin-on of the SAS Viya CLI(sas-viya). With both methods, it is the transfer service that does the work of creating a transfer package.
For Export, there are two steps involved.
Firstly, Export the content, this step creates a transfer package and stores it in the Infrastructure Data Server.
sas-viya --output text transfer export -u /folders/folders/$folderid
Secondly, Download the transfer package to a transfer package JSON file (the value for packageid is generated by the export step).
sas-viya transfer download -f /tmp/viya4packages/SalesContent.json --id $packageid
To Import, there are also two steps involved,
Firstly, Upload the transfer package file to a transfer package stored in the Infrastructure Data Server
sas-viya --output json transfer upload --file /tmp/viya4packages/SalesContent.json
Secondly, Import the uploaded transfer package (the value for packageid is generated by the export step).
sas-viya --output text transfer import --id $packageid
With SAS Environment Manager the two steps are executed via a single menu choice when export or import is selected, but the same thing happens behind the scenes. Using the sas-viya CLI you perform the two steps in order. The process with the CLI can be automated using bash scripting and 3rd-party tools like JQ as described in this post.
As you can see both import and export create packages that are stored in the Viya Infrastructure Data server. SAS Viya does not provide a user-interface to manage the packages stored in the Infrastructure Data Server. However, a Viya Administrator still has to manage this content, and to do that they need to understand what packages are available, what is in each package so that they can perform tasks like deleting or archiving old packages. Fortunately, these tasks can be (mostly) performed using the transfer plugin of the SAS Viya CLI.
What Packages exist in the Infrastructure Data Server
Using the transfer plugin of the sas-viya CLI you can list all packages stored in the Infrastructure Data Server.
sas-viya -y --output text transfer list
Select any image to see a larger version.
Mobile users: To view the images, select the "Full" version at the bottom of the page.
The list command allows the subsetting of the packages using the exact name of the package or a specified filter. To list packages with the name HRWorkinProgress you can run a command like this:
sas-viya -y --output text transfer list -n HRWorkinProgress
The filter can be useful and allows you to specify more complex queries. The filter syntax is the standard SAS REST API query syntax documented here.
For example, to list all packages that were created after a certain date by the user geladm.
sas-viya -y --output text transfer list -f "and(gt(creationTimeStamp,2022-10-06T00:00:00Z),eq(createdBy,'geladm'))"
As you can see from the output of the list command the name of a transfer package is not unique. It is the id field that uniquely identifies a package stored in the Infrastructure Data Server. This can be problematic for an Administrator trying to identify packages for download or deletion. In the command below we look for all packages where the name starts with the string Sales.
sas-viya -y --output text transfer list -f "and(startsWith(name,'Sales'),eq(createdBy,'geladm'))"
You can see from the output that we definitely have duplicates with regard to the “name”. Reviewing the other fields, the ResourceCount (number of objects in the package) column indicates that some of the packages may include the same content, but the CreateTimeStamp indicates they were created at different times.
Naming Transfer Packages
When a package is created using the CLI either via a transfer export or transfer upload you can, optionally, give the package a name (imports and exports using EV get a default name). It is an excellent practice to provide a name that will help you identify the package. You can name it with anything that makes sense and will help you to identify it at a later date. Some options are the folder path, name or id of the item you are exporting, a DateTime stamp of when you exported, or you could have a naming convention that includes a version number
In this example, we export a folder and name the package with the folder path and a version number. This will make it easy to identify the package at a later date.
sas-viya --output json transfer export -u /folders/folders/$folderid --name "/gelcontent/GELCorp/Sales_V2"
View the content that is in a Transfer Package
At some point, we will need to look inside a package and see what content is included. In the SAS Environment Manager user intercace when you open a package to import it you can see the content.
With the transfer CLI, using the id of a package we can look at what is included in a package in a variety of formats.
To see what is in a package we first need to retrieve the package id. This command returns the package id of the package we named "/gelcontent/GELCorp/Sales_V2". It use the JQ tool to parse the returned JSON and extract the id.
packageid=$(sas-viya --output json transfer list --name "/gelcontent/GELCorp/Sales_V2" | jq -r '.items[]["id"]')
Using the package id and the show command of the transfer CLI we can list the details of the content stored in the transfer package. Firstly, --output text and --details we can list individual items.
sas-viya --output text transfer show --details --id ${packageid}
Using --output text and --tree we get a nice picture of how the package content appears in the Viya folder structure.
sas-viya --output text transfer show --id ${packageid} --tree
Deleting Transfer Packages from the Infrastructure Data Server
Packages can consume space in the infrastructure data server as was mentioned before there is no UI to manage them. If we want to delete them we need to use the CLI. The transfer plug-in has a delete command that accepts the id and URI and will delete one package at a time
<span style="font-weight: 400;">sas-viya --output text transfer delete --id</span> ${packageid}
Sas-viya transfer delete is not the most convenient command because we can only delete one transfer package at a time, and we need to know the URI or ID
The pyvyatools are a companion to the sas-viya CLI designed to allow the fast development of tools that extend the functionality of the CLI. There are two tools that can help with the management of transfer packages.
Listtransferpackages.py allows an administrator to query what transfer packages are stored in the infrastructure data server. It is a bit more flexible in querying, sorting, and in the style of output provided than sas-viya transfer list. For example, we can list in CSV format all transfer packages in the Infrastructure Data Server that have “Sales” in the package name and return the packages in descending order of modified date so that the most recently created package is listed first.
./listtransferpackages.py -o csv -n /gelcontent/GELCorp/Sales -s modifiedTimeStamp -so descending
Deletetransferpackages.py allows the administrator to delete multiple packages with one execution. You can delete packages that match a query based on the user that last modified the package, the age of the package, and the name of the package. For example: Delete all transfer packages modified by geladm and with “Sales” in the name that are a day old or more.
./deletetransferpackages.py -m geladm -n /gelcontent/GELCorp/Sales -d 1 -do older
Summary
SAS Viya transfer packages are created and stored in the SAS Viya Infrastructure Data Server when content is exported and imported. SAS Viya Transfer Packages size varies based on the volume of content in a package. In a long-running SAS Viya environment packages can start to consume significant space in the Infrastructure Data Server. Using sas-viya CLI and pyviyatools the SAS Administrator can accurately identify packages for download and deletion. I hope you found this post useful.
Find more articles from SAS Global Enablement and Learning here.
