Importing User-defined Formats to CAS in the SAS Viya Platform
User-defined formats are long-term SAS features that are very powerful and often used by SAS programmers and developers. User-defined formats can be used to change how data is displayed and can provide a means to do data conversion and even table lookups. When SAS introduced CAS as a new in-memory analytical engine it included support for using SAS formats. However, there are significant differences in the way CAS treats formats to the way the SAS Programming run-time servers (Base SAS) do. With the release of the SAS Viya platform running on Kubernetes and the focus on migration from SAS 9.4 to Viya, it is worth taking another look at how you make User-Defined formats available to CAS.
User-defined formats in CAS Overview
User-defined formats in the SAS Programming run-time (9.4 or Viya Compute) are stored and read from SAS catalogs inside a SAS Library. In Viya the CAS server cannot read SAS catalogs. To use formats the CAS server requires formats to be stored in a CAS Format library inside a CAS Library.
To make formats available to the CAS Server SAS catalogs must be converted to a CAS Format Library (sashdat file). The CAS Format Library must be:
- promoted to global
- added to the Formats path-based CAS library
- added to the CAS server format search path.
NOTE: The CAS format library is a .sashdat file. However, the .sashdat files are in a different format from the .sashdat files that contain data.
Let us look at the out-of-the-box configuration for user-defined formats in Viya 4. The standard install sets up a global path-based CAS library named Formats. The CAS library is physically located on the cas-default-data CAS persistent volume and is mounted into the CAS Controller at /cas/data/caslibs/formats/. In Viya 4, unlike Viya 3.x, there are no default CAS format libraries available in the CAS library.
How to make user-defined formats available
There are two levels of access to format administration in Viya.
A SAS Viya Administrator can manage formats in existing CAS format libraries including:
- Create new formats
- Edit existing formats
- Copy and delete formats
- Import formats from an:
- Item store file accessible from the CAS controller
- SAS catalog file accessible from the machine that runs the compute service
- sashdat file accessible from the CAS Controller
In addition to all the above functionality, a SAS Viya administrator that is also a CAS Super User can create and delete CAS format libraries and manage the CAS format search path.
File Types available for import
The three types of files from which we can import formats are a SAS catalog, a SAS item store, and .sashdat file. Let us look at the differences.
SAS Catalog
Customers migrating from SAS 9.4 will likely import their formats from SAS Catalogs. LINUX catalogs can be imported directly, catalogs from other operating systems must be exported using PROC CPORT and imported using PROC CIMPORT. A SAS catalog file must be encoded in UTF-8 to import. If your catalog is already UTF-8 then you can import it directly. If your catalog is not UTF-8 then you can convert it to UTF-8 (CPORT and CIMPORT), or convert it to a SAS item store and import the item store.
SAS Catalog files can be imported from locations accessible to the Viya Compute Server. The files must have the extension .sas7bcat.
SAS Item store file
A SAS item store is a file that can contain multiple format libraries. The FMTC2ITM procedure can copy the formats in one or more format catalog to a single item store physical file. The item store can be convenient as it allows you to package server SAS catalogs into a single file for import. In addition, PROC FMT2CITM can convert natively encoded catalogs to UTF-8(the default encoding on Viya). The example code below shows two SAS catalogs being added to an item store.
/* create itemstore with all formats in one file*/
proc fmtc2itm catalog=(sformats.formats hformats.formats) encoding=wlatin1 print locale itemstore="c:\temp\cport\itemstore\allformats.itemstore";
SAS Item store files can be imported from locations accessible to the Viya CAS Controller. The files must have the extension .itemstore (or no extension for import with the sas-viya CLI)
NOTE: the FMTC2ITM procedure is available in SAS 9.4M3 and later.
SASHDAT file
For customers already on Viya, if you have an existing CAS format library stored in a .sashdat file, it can be imported to a different Viya environment. SASHDAT files can be imported from locations accessible to the Viya CAS Controller.
Making files available for import to Viya
Prior to import the files containing the formats must be copied to a location accessible to either CAS or Compute. This can be a little bit more complex in Viya 4. In most cases, a Viya 4 environment will have a shared storage location mounted from a location outside the cluster to the CAS and Compute servers. This is the location where you can copy the files that contain your formats. In addition, this location must be in the allowlist for the selected server. For more details on this topic see this post: Accessing path-based data from CAS in Viya 4
In rare situations where you do not have shared storage, a Kubernetes Administrator can copy files to the CAS controller using the Kubernetes command-line interface (kubectl).
kubectl cp "/shared/gelcontent/gelcorp/cport/itemstore/allformats.itemstore" sas-cas-server-default-controller:/tmp -c sas-cas-serverInterface to import user-defined formats
There are three interfaces available to import user-defined formats. All the interfaces support the three types of files.
SAS Environment Manager
SAS Environment manager provides a visual user interface to manage formats. A Viya Administrator who is a CAS Super User can create and manage format libraries and import formats. The details are documented at SAS Help Center: Managing User-Defined Formats in SAS Viya Platform: Manage Format Libraries
A Viya Administrator (not a CAS Super User) can create new formats, edit existing formats, and import formats to existing CAS format libraries. The details are documented at SAS Help Center: Managing User-Defined Formats in SAS Viya Platform: Manage Formats
NOTE: the file extension of the source path option can be .sashdat, .sas7bcat, or item store. The file extension triggers what type of import occurs.
CAS plug-in to the sas-viya command line interface
The CAS plug-in to the sas-viya command allows the administrator to create format libraries, update the format search order, and import formats. The command below performs all these tasks in one step.
./sas-viya cas format-libraries create --server cas-shared-default --format-library hrformats --search-order prepend --source-path /gelcontent/gelcorp/hr/formats/formats.sas7bcat --su
NOTE: the file extension of the --source-path option can be .sashdat or .sas7bcat any other file extension is treated as an item store. For more details on all the tasks you can perform with the sas-viya command check out SAS Help Center: CLI Examples: Formats
SAS Code
You can also use SAS Code to manage user-defined formats in CAS. For more details check out this documentation at SASHELP: Using User-Defined Formats in SAS Cloud Analytic Services: Migrate User-Defined Formats fro...
CAS Format Library persistence
In previous versions of Viya, there were additional steps required to persist CAS format libraries across CAS server restarts. In Viya 4 there are no added steps needed. CAS format libraries persist across CAS server restarts by default. For more details on this change check out this post. SAS Viya CAS Format library persistence
Summary
If a CAS table has columns that reference user-defined formats, the formats must be made available to the CAS server. CAS cannot directly read SAS catalogs that store user-defined formats. They must be converted to a CAS format library and stored in a .sashdat file in the Formats path-based CASLIB. A Viya administrator can perform the steps to import formats using SAS Environment Manager, the sas-viya command-line interface, or SAS code.
For more information.
CAS Documentation: User-Defined Formats in CAS
Viya Platform Administration Guide: Managing User-Defined Formats in SAS Viya Platform
Viya Administration Guide: Formats (CAS) CLI Examples
Thanks to my colleagues @ScottMcCauley, @ChristianaLycan, and Gregor Hermann for their input on this post. Look for an upcoming post from Gregor on using user-defined formats with VA on Viya.
