Loading SAS Data with Formats to Object Storage Parquet

1 Like

Parquet's columnar structure and intelligent compression scheme make it the go-to choice for the latest analytic developments like SAS' DuckDB integration. So let's convert our on-prem SAS datasets to Parquet on their way to cloud object storage like AWS, ADLS, and GCS for use in Viya....

SAS Formats?

But what if your data and processing rely on SAS formats? Currently SAS cannot embed SAS Format definitions into Parquet files like it can on other data platforms. So, how can we utilize Parquet on object storage and still have our SAS formats?!?!?

Why Formats?

SAS formats are the intelligent storage compression scheme that predates Parquet by years. With a user-defined format, SAS can simultaneously treat a field as both a lookup code/ID as well as a long-form description. It's like having your transaction table and reference tables all in one or Facts combined with Dimensions without the overhead of repeating those long descriptions over and over again. Formatting data for presentation was their initial purpose but SAS formats quickly became a space saving technique, a high performance joining tool, a sub-setting tool, and much more.

Load On-Prem SAS to Object Storage Parquet

First, let's tackle the easy part. Loading SAS datasets (or any data SAS can read) to object storage is easy with the Parquet libname. You simply define a library with connectivity to your object storage bucket and write the table you want to it. I'll use code to demonstrate since it captures the ease of connecting as well as the natural integration but you could use either of Viya's graphical interfaces, Enterprise Guide or SAS Studio Flows.

*** Create SAS Library to ADLS2 Blob Storage ***;
options azuretenantid="myTenantID";

libname myBlob parquet "/myBucketPath" 
    storage_platform = "adls"
    storage_account_name = "myADLSStorageAccount"
    storage_file_system = "myADLSFileSystem"
    storage_application_id = "myClientID"
    storage_client_secret = "myClientSecret";

*** Write the On-Premises Customer Table to ADLS2 Blob Storage ***;
data myBlob.customer;
set onPremLb.customer;
run;

Once we define the libname to the object storage container, Azure ADLS2 in this case, all we have to do is write our on-prem SAS dataset to it. In this example, we do it with a SAS DATA step but we could use SQL, Tabulate, SORT, whatever....

Extract and Load the Format Information

As discussed, any SAS format information on the table is lost when loaded to Parquet. Only the raw data, the codes and IDs, will be written to the parquet file. We could, of course, explicitly render the formatted values using PUT functions but let's do better, save space, and improve efficiency by keeping the formats.

We'll need two separate components, the format catalog which contains the codes-to-long-form-descriptions mapping as well as the table's field format assignments, as in which fields use which formats.

Loading the Format Catalog to Object Storage

If our operating systems are the same we could just load the format catalog as is, but let's do it the most general way possible to cover the most scenarios. We'll extract the format contents and load them as Parquet as well.

*** Extract the format catalog to Parquet on ADLS2 ***;
proc format library=library.formats 
            cntlout=myBlob.customFormats;
run;

Using the myBlob library defined previously, loading the on-premises formats to object storage is only a two line job. We could refine the code to extract only the formats we need and we could also use change-data-capture techniques since we are storing the formats as table data.

Here we show the formats coming from the default format catalog, library.formats, but your formats may reside in a different location. To locate your environment's format catalogs, check the FMTSEARCH option setting.

Loading the Table's Format Assignments

The last puzzle piece is our table's column format assignments. We can extract these from the SAS dictionary tables and load them to ADLS2 as well with a simple SQL statement.

*** Extract the format catalog to Parquet on ADLS2 ***;
proc sql noprint;
    create table myBlob.customer_formats as
    select name as column_name, format 
    from dictionary.columns 
    where libname="ONPREM"
      and memname="CUSTOMER";
quit;

This results in a table which simply identifies which fields use which formats. Our sample table has 11 fields, 4 of which use formats as shown.

Select any image to see a larger version.
Mobile users: To view the images, select the "Full" version at the bottom of the page.

Next Steps

With only four SAS statements, we've loaded our on-prem SAS dataset along with its formats and column format assignments to cheap object storage in the highly efficient Parquet format.

In my next post, I'll demonstrate how to use these files directly in Compute (base SAS) with minimal code modifications as well as to load CAS for Viya's advanced analytic and visual capabilities.

Find more articles from SAS Global Enablement and Learning here.

MAGG · ‎07-08-2026

Is this possible with EG 8.6 ? I'am under the impression SAS 9.4 had no no built-in Parquet LIBNAME engine.

I'am aware SAS In-Database Code Accelerator. But the above example explicitly an on-prem SAS dataset .

Quentin · ‎07-08-2026

I don't think you can get it to work in 9.4 without external tools (python etc), unfortunately. I thought I head at SAS Innovate a couple years ago that the Duck DB engine would come to 9.4, but, sadly, apparently this is not the case: https://communities.sas.com/t5/SAS-Data-Management/SAS-Connection-to-Duckdb/td-p/949322 .

That said, PROC PYTHON is coming to 9.4M10 (https://communities.sas.com/t5/SAS-Innovate-Presentations/SAS-9-4-M10-Continuing-Support-and-Roadmap... ) so that should provide a way to read Parquet, and lots of other stuff, without leaving your SAS session. You can probably do it now in earlier maintenance releases using PROC FCMP interface to python.