topic Re: Connecting SAS to a Hadoop based data lake in SAS Programming

Connecting SAS to a Hadoop based data lake

juanvg1972 — Thu, 14 Jun 2018 20:52:33 GMT

Hi,

I am working with a SAS Base in a server and I need to connect to a Hadoop based data lake to get some data from HDFS.

I am using SAS/ACCESS to Hadoop and using a libname to Hive, then I run a query in Hadoop. This type of connection works but It's very slow,.It takes more than an hour tu run a Hive query that get 10.000 rows from a 10 millions rows file.

Is there any other way to ge data from Hadoop in SAS?, can I connect to Impala vía libname? in the data lake I have algo Spark installed.

Any advice will be greatly appreciated

Thanks in advance

Re: Connecting SAS to a Hadoop based data lake

ChrisNZ — Fri, 15 Jun 2018 02:29:42 GMT

As I recall, Hive is very slow for querying and is best used to write data, while Impala is best used to query data.

Re: Connecting SAS to a Hadoop based data lake

juanvg1972 — Fri, 15 Jun 2018 08:00:52 GMT

Yes, I know it, but how can I connect Impala from SAS?

Thanks

Re: Connecting SAS to a Hadoop based data lake

juanvg1972 — Fri, 15 Jun 2018 21:35:17 GMT

I have found this:

https://www.sas.com/en_us/software/access-interface-impala.html

I have other question:

Is there any way of execute SAS code (data steps, proc sql and statistycal procs) in a Hadoop in a distrbuted way?

Is there any module to do this?. If I transforma this code to proc ds2 o proc fedsql it will be possible?

Thanks in advance

Re: Connecting SAS to a Hadoop based data lake

ChrisNZ — Mon, 18 Jun 2018 06:03:05 GMT

1. SAS/ACCESS or ODBC ( https://www.cloudera.com/downloads/connectors/impala/odbc/2-5-37.html ) allow you to use Impala from SAS

2. You need SAS Embedded Process to execute SAS code within the Hadoop cluster.

Otherwise you are limited to SQL-type queries.

Re: Connecting SAS to a Hadoop based data lake

sriramsahani — Wed, 20 Mar 2019 07:20:45 GMT

Hello,

Did you find a way to connect to Impala without using SAS Access to Impala, using the Hadoop engine ?

Regards,

Re: Connecting SAS to a Hadoop based data lake

ChrisNZ — Wed, 20 Mar 2019 08:21:26 GMT

The Hadoop engine connects to Hive.
You may be able to connect to Impala via ODBC?

Re: Connecting SAS to a Hadoop based data lake

sriramsahani — Wed, 20 Mar 2019 12:47:01 GMT

Thank you Chris for confirming