BookmarkSubscribeRSS Feed

Managing SAS Viya Transfer Packages

Started ‎10-17-2022 by
Modified ‎11-07-2022 by
Views 1,699

In SAS Viya we can move content between environments by creating transfer packages. Transfer packages are created in the SAS Viya Infrastructure Data server and can be downloaded to a transfer package JSON file on the file system for movement between environments. To manage transfer package files in the file system, you can copy, move and delete the packages as you would any file. However, for the transfer packages stored in the Infrastructure Data server management is a little more complex. In this post, I will look at managing the transfer packages that are created and saved in the Infrastructure Data Server during the import and export of Viya Content.

 

Export and import of content in Viya can be performed using SAS Environment Manager or the transfer plugin-on of the SAS Viya CLI(sas-viya). With both methods, it is the transfer service that does the work of creating a transfer package.

 

For Export, there are two steps involved.

 

Firstly, Export the content, this step creates a transfer package and stores it in the Infrastructure Data Server.

 

sas-viya --output text transfer export -u /folders/folders/$folderid 

 

Secondly, Download the transfer package to a transfer package JSON file (the value for packageid is generated by the export step).

 

sas-viya transfer download -f /tmp/viya4packages/SalesContent.json --id $packageid

 

To Import, there are also two steps involved,

 

Firstly, Upload the transfer package file to a transfer package stored in the Infrastructure Data Server

 

sas-viya --output json transfer upload --file /tmp/viya4packages/SalesContent.json

 

Secondly, Import the uploaded transfer package (the value for packageid is generated by the export step).

 

sas-viya --output text transfer import --id $packageid

 

With SAS Environment Manager the two steps are executed via a single menu choice when export or import is selected, but the same thing happens behind the scenes. Using the sas-viya CLI you perform the two steps in order. The process with the CLI can be automated using bash scripting and 3rd-party tools like JQ as described in this post.

 

As you can see both import and export create packages that are stored in the Viya Infrastructure Data server.  SAS Viya does not provide a user-interface to manage the packages stored in the Infrastructure Data Server. However, a Viya Administrator still has to manage this content, and to do that they need to understand what packages are available, what is in each package so that they can perform tasks like deleting or archiving old packages. Fortunately, these tasks can be (mostly) performed using the transfer plugin of the SAS Viya CLI.

 

What Packages exist in the Infrastructure Data Server

 

Using the transfer plugin of the sas-viya CLI you can list all packages stored in the Infrastructure Data Server.

 

sas-viya -y --output text transfer list

 

gn_transferpackage_001-1.png

Select any image to see a larger version.
Mobile users: To view the images, select the "Full" version at the bottom of the page.

 

The list command allows the subsetting of the packages using the exact name of the package or a specified filter. To list packages with the name HRWorkinProgress you can run a command like this:

 

sas-viya -y --output text transfer list -n HRWorkinProgress

 

gn_transferpackage_002.png

 

The filter can be useful and allows you to specify more complex queries. The filter syntax is the standard SAS REST API query syntax documented here.

 

For example, to list all packages that were created after a certain date by the user geladm.

 

sas-viya -y --output text transfer list -f "and(gt(creationTimeStamp,2022-10-06T00:00:00Z),eq(createdBy,'geladm'))"

 

gn_transferpackage_003-1.png

 

As you can see from the output of the list command the name of a transfer package is not unique. It is the id field that uniquely identifies a package stored in the Infrastructure Data Server. This can be problematic for an Administrator trying to identify packages for download or deletion. In the command below we look for all packages where the name starts with the string Sales.

 

sas-viya -y --output text transfer list -f "and(startsWith(name,'Sales'),eq(createdBy,'geladm'))"

 

gn_transferpackage_004.png

 

You can see from the output that we definitely have duplicates with regard to the “name”. Reviewing the other fields, the ResourceCount (number of objects in the package) column indicates that some of the packages may include the same content, but the CreateTimeStamp indicates they were created at different times.

 

Naming Transfer Packages

 

When a package is created using the CLI either via a transfer export or transfer upload you can, optionally, give the package a name (imports and exports using EV get a default name). It is an excellent practice to provide a name that will help you identify the package. You can name it with anything that makes sense and will help you to identify it at a later date. Some options are the folder path, name or id of the item you are exporting, a DateTime stamp of when you exported, or you could have a naming convention that includes a version number

 

In this example, we export a folder and name the package with the folder path and a version number. This will make it easy to identify the package at a later date.

 

sas-viya --output json transfer export -u /folders/folders/$folderid --name "/gelcontent/GELCorp/Sales_V2"


View the content that is in a Transfer Package

 

At some point, we will need to look inside a package and see what content is included. In the SAS Environment Manager user intercace when you open a package to import it you can see the content.  

 

gn_transferpackage_005-1.png

 

With the transfer CLI, using the id of a package we can look at what is included in a package in a variety of formats.

 

To see what is in a package we first need to retrieve the package id. This command returns the package id of the package we named  "/gelcontent/GELCorp/Sales_V2". It use the JQ tool to parse the returned JSON and extract the id.

 

packageid=$(sas-viya --output json transfer list --name  "/gelcontent/GELCorp/Sales_V2" | jq -r '.items[]["id"]')

 

Using the package id and the show command of the transfer CLI we can list the details of the content stored in the transfer package. Firstly, --output text and --details we can list individual items.

 

sas-viya --output text transfer show --details --id ${packageid}

 

gn_transferpackage_006.png

 

Using --output text and --tree we get a nice picture of how the package content appears in the Viya folder structure.

 

sas-viya --output text transfer show  --id ${packageid} --tree

 

gn_transferpackage_007.png

 

Deleting Transfer Packages from the Infrastructure Data Server

 

Packages can consume space in the infrastructure data server as was mentioned before there is no UI to manage them. If we want to delete them we need to use the CLI. The transfer plug-in has a delete command that accepts the id and URI and will delete one package at a time

 

<span style="font-weight: 400;">sas-viya --output text transfer delete --id</span> ${packageid}

 

Sas-viya transfer delete is not the most convenient command because we can only delete one transfer package at a time, and we need to know the URI or ID

 

The pyvyatools are a companion to the sas-viya CLI designed to allow the fast development of tools that extend the functionality of the CLI. There are two tools that can help with the management of transfer packages.

 

Listtransferpackages.py allows an administrator to query what transfer packages are stored in the infrastructure data server. It is a bit more flexible in querying, sorting, and in the style of output provided than sas-viya transfer list. For example, we can list in CSV format all transfer packages in the Infrastructure Data Server that have “Sales” in the package name and return the packages in descending order of modified date so that the most recently created package is listed first.

 

./listtransferpackages.py -o csv -n /gelcontent/GELCorp/Sales -s modifiedTimeStamp -so descending

 

gn_transferpackages_008.png

 

Deletetransferpackages.py allows the administrator to delete multiple packages with one execution. You can delete packages that match a query based on the user that last modified the package, the age of the package, and the name of the package. For example: Delete all transfer packages modified by geladm and with “Sales” in the name that are a day old or more.

 

./deletetransferpackages.py -m geladm -n /gelcontent/GELCorp/Sales -d 1 -do older

gn_transferpackages_009.png

 

Summary

 

SAS Viya transfer packages are created and stored in the SAS Viya Infrastructure Data Server when content is exported and imported.  SAS Viya Transfer Packages size varies based on the volume of content in a package. In a long-running SAS Viya environment packages can start to consume significant space in the Infrastructure Data Server. Using sas-viya CLI and pyviyatools the SAS Administrator can accurately identify packages for download and deletion. I hope you found this post useful.

 

Find more articles from SAS Global Enablement and Learning here.

Version history
Last update:
‎11-07-2022 10:39 AM
Updated by:
Contributors

SAS INNOVATE 2024

Innovate_SAS_Blue.png

Registration is open! SAS is returning to Vegas for an AI and analytics experience like no other! Whether you're an executive, manager, end user or SAS partner, SAS Innovate is designed for everyone on your team. Register for just $495 by 12/31/2023.

If you are interested in speaking, there is still time to submit a session idea. More details are posted on the website. 

Register now!

Free course: Data Literacy Essentials

Data Literacy is for all, even absolute beginners. Jump on board with this free e-learning  and boost your career prospects.

Get Started