topic Re: Reading Parquet-Files in SAS 9.4 in SAS Data Management

Reading Parquet-Files in SAS 9.4

SvenWagner — Fri, 05 Jul 2019 06:02:45 GMT

Hi,

I use SAS 9.4 TS Level 1M3. Does anyone know a solution for reading Parquet files (https://en.wikipedia.org/wiki/Apache_Parquet ) by using SAS without having SAS/ACCESS Interface to Hadoop?

Thanks a lot, Sven

Re: Reading Parquet-Files in SAS 9.4

Patrick — Fri, 05 Jul 2019 06:54:22 GMT

So how do you plan to connect to Hadoop to get access to the file in first place?

Re: Reading Parquet-Files in SAS 9.4

SvenWagner — Fri, 05 Jul 2019 07:11:47 GMT

The background is that a colleague uses Spark and Scala for producing data, and he stores his results within Parquet files. I think, his setup comprises a hive server but he does not use Hadoop.

So far, his recommendation is to use a Hive or a MapR Drill ODBC driver to get access to the data via SAS/ACCESS Interface to ODBC. And my question is, if someone has experience in doing so, or if someone knows an alternative (without having SAS Hadoop).

Sorry for the amateurish explanation, but I am not a techie at all, and I do not have a good understanding of the technical infrastructure.

Re: Reading Parquet-Files in SAS 9.4

Patrick — Fri, 05 Jul 2019 10:53:55 GMT

You need either SAS/Access to Hadoop, SAS/Access to ODBC or SAS/Access to Impala

Do you have any of these interfaces licensed?

Re: Reading Parquet-Files in SAS 9.4

Reeza — Fri, 05 Jul 2019 15:11:46 GMT

Use the following to check your licenses, the information is in the log:

*show what's licensed;
proc setinit;run;

*what's installed;
proc product_status;

run;

Re: Reading Parquet-Files in SAS 9.4

SvenWagner — Mon, 08 Jul 2019 06:09:34 GMT

SAS/ACCESS Interface to ODBC is licensed and also installed, but neither the Hadoop package nor the Impala package.

Re: Reading Parquet-Files in SAS 9.4

vipinj765 — Fri, 11 Sep 2020 09:37:24 GMT

Hi,
I have local downloaded copy of parquet file on linux server. Instead of connecting to hadoop cluster, i want to read the local version. Is there any way to do this in SAS 9.4m6 ?

Re: Reading Parquet-Files in SAS 9.4

LinusH — Mon, 14 Sep 2020 11:49:26 GMT

The only (?) alternative to the ODBC concept is have the parquet file exported to a format that you local SAS session can access, e.g. csv.

Re: Reading Parquet-Files in SAS 9.4

LinusH — Mon, 14 Sep 2020 12:30:34 GMT

Just want to add, if you create a csv file, that hasn't to be moved to your local SAS server.
You can use FILENAME HADOOP to access the file in hdfs.
PROC HADOOP let you submit hdfs commands or exxecute pig programs which can come in handy when converting the parquet file.
None of these require a separate SAS/ACCESS licence.