Passing SAS Functions to Hadoop

[msd83]

Hi all,

I was wondering if anyone has any good links on what is pushed down to Hadoop when the Hadoop engine is used. I am using SAS 9.4.

Also, is the processing only carried out in the Hadoop cluster when PROC SQL is used?

Is anything done in the datastep?

Many thanks

[LinusH · Accepted solution]

List of functions supported by SAS/ACCESS is listed in the documentation:

http://support.sas.com/documentation/cdl/en/acreldb/66787/HTML/default/viewer.htm#p010gv8z4k5kvin1om...

The concept about processing in Hadoop is pretty much the same as any other RDBMS. This is quite clearly described in the documentation as well.

The same goes with the data step, which in short form is:

• Where-clause (any used function must be supported as described above)
• Implicit sorting if BY is used

Use options sastrace=',,,d' sastraceloc=SASLOG; to see what SQL the ACCESS engine issues against Hive.

[msd83]

Hi LunusH,

That is great, many thanks.

[LinusH]

Just found this about data step (pre production...):

SAS(R) 9.4 In-Database Products: User's Guide, Third Edition

[msd83]

Hi, that is really useful.

[JBailey]

Hi,

You can make SAS tell you which functions are eligible for implicit pass-through (to Hive). Here is an example (off the top of my head so it may just be close):

libname myhdp hadoop server=myHadoop sql_functions_copy=work.hadoop_functions;

This will create a SAS data set (in work) which lists the functions available for implicit pass-through. Keep in mind, implicit pass-through must be invoked in order to have them pass to Hadoop. It will also show you which Hive functions they map too.

When you are looking at your SASTRACE=',,,d' output, keep in mind it is more important that functions on a WHERE clause pass down. Post processing them (SAS processing the functions) could cause the entire contents of the Hive table to be returned.