topic Re: Passing SAS Functions to Hadoop in SAS Data Management

Passing SAS Functions to Hadoop

msd83 — Tue, 06 May 2014 10:58:26 GMT

Hi all,

I was wondering if anyone has any good links on what is pushed down to Hadoop when the Hadoop engine is used. I am using SAS 9.4.

Also, is the processing only carried out in the Hadoop cluster when PROC SQL is used?

Is anything done in the datastep?

Many thanks

Re: Passing SAS Functions to Hadoop

LinusH — Tue, 06 May 2014 12:39:38 GMT

List of functions supported by SAS/ACCESS is listed in the documentation:

http://support.sas.com/documentation/cdl/en/acreldb/66787/HTML/default/viewer.htm#p010gv8z4k5kvin1ombbmqdpumhd.htm

The concept about processing in Hadoop is pretty much the same as any other RDBMS. This is quite clearly described in the documentation as well.

The same goes with the data step, which in short form is:

Where-clause (any used function must be supported as described above)
Implicit sorting if BY is used

Use options sastrace=',,,d' sastraceloc=SASLOG; to see what SQL the ACCESS engine issues against Hive.

Re: Passing SAS Functions to Hadoop

msd83 — Tue, 06 May 2014 13:04:57 GMT

Hi LunusH,

That is great, many thanks.

Re: Passing SAS Functions to Hadoop

LinusH — Wed, 07 May 2014 07:30:13 GMT

Just found this about data step (pre production...):

SAS(R) 9.4 In-Database Products: User's Guide, Third Edition

Re: Passing SAS Functions to Hadoop

msd83 — Wed, 07 May 2014 09:05:51 GMT

Hi, that is really useful.

Re: Passing SAS Functions to Hadoop

JBailey — Mon, 12 May 2014 19:43:56 GMT

Hi,

You can make SAS tell you which functions are eligible for implicit pass-through (to Hive). Here is an example (off the top of my head so it may just be close):

libname myhdp hadoop server=myHadoop sql_functions_copy=work.hadoop_functions;

This will create a SAS data set (in work) which lists the functions available for implicit pass-through. Keep in mind, implicit pass-through must be invoked in order to have them pass to Hadoop. It will also show you which Hive functions they map too.

When you are looking at your SASTRACE=',,,d' output, keep in mind it is more important that functions on a WHERE clause pass down. Post processing them (SAS processing the functions) could cause the entire contents of the Hive table to be returned.