topic Re: How can i export my data in parquet format in SAS Programming

How can i export my data in parquet format

umarigan — Wed, 06 Sep 2023 08:25:04 GMT

I am trying to download a data but cant because it is big(195MB), so i am checking alternative to do this task. one way to downsize data size is using parquet format so I am wondering if there is a way to convert and download data with code?

Re: How can i export my data in parquet format

LinusH — Wed, 06 Sep 2023 08:52:37 GMT

We need some more information before we can give any valuable advice.

Were is the data now, in what format?

Download, using what method?

SAS version/product, on server and client.

Re: How can i export my data in parquet format

sbxkoenk — Wed, 06 Sep 2023 08:55:58 GMT

You need a recent enough SAS Viya 4.

SAS Viya and Parquet files – additional features
Started ‎01-26-2023 | Modified ‎01-26-2023
https://communities.sas.com/t5/SAS-Communities-Library/SAS-Viya-and-Parquet-files-additional-features/ta-p/855800

BR, Koen

Re: How can i export my data in parquet format

umarigan — Wed, 06 Sep 2023 09:27:48 GMT

I am a bit new to this but my data can be seen on manage data section, I write a simple code to display some output as follow:

/* Print the first 10 observations **/
cas casauto;
caslib _ALL_ assign;
options casdatalimit=all;
options compress=yes;
%let path= /caslibs/public;
libname public "&path";

proc print data=PUBLIC.mydata(obs=5);
run;

to download data, in manage data tab in up left 3 dot section i simply select download table and it downloads a csv file with error saying "Table to stream failed with details: initial exception: 'Read timed out', converted exception: 'Read timed out'." along with name of columns in table

sas version: 3.5

Re: How can i export my data in parquet format

LinusH — Wed, 06 Sep 2023 12:13:40 GMT

It seems like it's possible to load Parquet in Viya 3.5 from a CASLIB:

SAS Help Center: Parquet Data Sets

Re: How can i export my data in parquet format

fierceanalytics — Mon, 20 Nov 2023 14:55:05 GMT

Hello,

I don't have Viya. I do have Jupyter notebook. Just run Pd. read_parquet to read in your parquest file. Write out to CSV and then use SAS proc import. to read it back in. I run batch to process many GB size. CSV is your best bridge between SAS and open source data files. It has a distinctive speed edge over SAS Hadoop Access. I once tested Viya and felt it still has too many layers in the background like 'old SAS'. It may be optimizing something, but for 195 MB? you don't need that elaboration. You just need straight shot, simple and fast.

Jia