topic Re: Saving an In-Memory CAS Table as a Single Parquet File in SAS Procedures

Saving an In-Memory CAS Table as a Single Parquet File

alisio_meneses — Fri, 05 Apr 2024 17:44:02 GMT

Hi there,

I see 'PROC CASUTIL SAVE' can write an in memory CAS table to multiple parquet files stored inside a CASLIB folder. I wonder if it is possible to do it into single file parquet file.

Is it? if so, how?

Env. Info: SAS Viya 3.5 running on linux with multiple cas workers.

Thank you!

Re: Saving an In-Memory CAS Table as a Single Parquet File

sbxkoenk — Sat, 06 Apr 2024 09:07:34 GMT

... to multiple parquet files??

Strange.
By default, one CAS-table goes into one .parquet file.
Why would it be split across multiple *.parquet files?

proc casutil;
   save casdata="carsInMemory" casout="carsFile.parquet";
run;

See here :

SAS® 9.4 and SAS® Viya® 3.5 Programming Documentation
CAS User’s Guide
Parquet Data Sets

https://go.documentation.sas.com/doc/en/pgmsascdc/9.4_3.5/casref/p0u5p2nvqu04gfn1w3zaohdfcoys.htm#n0da8b2zq12h27n1d2nmvwxf53il

Koen

Re: Saving an In-Memory CAS Table as a Single Parquet File

Patrick — Sun, 07 Apr 2024 05:01:15 GMT

@sbxkoenk In my environment with a recent Viya 4 version and 4 worker nodes the parquet file gets created in chunks (=multiple files) all stored under a folder with the name of the parquet file that had been provided as value to parameter casout.

I couldn't find a way to only create a single file using Proc Casutil. I do believe that chunks are required for full support of parallelism.

I could create a single parquet file via client side (compute) processing using a data step.

%let sessref=MySess;
%if %sysfunc(sessfound(&sessref)) %then
  %do;
    cas mySess terminate;
  %end;
cas &sessref cassessopts=(caslib="casuser" /*metrics=True*/);
libname casuser cas;
options fullstimer msglevel=i ps=max;

data casuser.class;
  set sashelp.class;
run;


libname comp_pq parquet "&_userhome";

data comp_pq.class_datastep;
  set casuser.class;
run;

caslib cas_pq path="&_userhome" datasource=(srctype="path");
proc casutil;
  save casdata="class" incaslib="casuser" casout="class_casutil.parquet" replace; 
quit;

/* cas mySess terminate; */

Re: Saving an In-Memory CAS Table as a Single Parquet File

sbxkoenk — Sun, 07 Apr 2024 09:30:42 GMT

Sorry. I had better not said anything. Too little experience with Parquet files. ☹️😞

Maybe @UttamKumar can help.

SAS Viya and Parquet files – additional features
Started ‎01-26-2023 | Modified ‎01-26-2023

by UttamKumar
https://communities.sas.com/t5/SAS-Communities-Library/SAS-Viya-and-Parquet-files-additional-features/ta-p/855800

Koen

Re: Saving an In-Memory CAS Table as a Single Parquet File

alisio_meneses — Mon, 08 Apr 2024 11:46:32 GMT

hello, thanks for the reply. Using save generates a single folder named <tablename.parquet> with multiple parquet files inside. I guess thats for optimization purposes. Not sure.