vivshri Tracker

Error getting "The HADOOP engine cannot be found." while using libname

vivshri — Wed, 11 Feb 2015 11:01:04 GMT

Hi,

I am trying to connect SAS 9.3 over Windows 2008 64 Bit to Hadoop 2.4.0 Apache distribution. I can run the following command successfully

proc hadoop username='hadoop' password='mypassword' verbose;

hdfs mkdir='/tmp/d';

run;

But when I try to use libname, I get the following error.

proc sql;

connect to hadoop (user="myuser" pw=XXXXXXX server=hxpduped);

ERROR: The HADOOP engine cannot be found.

ERROR: A Connection to the hadoop DBMS is not currently supported, or is not installed at your

site.

libname hdp hadoop BL_PORT=9001 server="xx.xx.xx.xx" port=10000 subprotocol=hive2

! schema=default

user=hadoop password=XXXXXXXX verbose;

ERROR: The HADOOP engine cannot be found.

ERROR: Error in the LIBNAME statement.

Is something missing in the jar files? I have setup SAS_HADOOP_JAR_PATH and SAS_HADOOP_CONFIG.

Re: Unable to run Hadoop command

vivshri — Wed, 11 Feb 2015 10:46:39 GMT

Thanks Linus, Just to update for future readers, the error is resolved. It seems the SAS_HADOOP_JAR_PATH was not readable. After discussing with SAS support I put that in all user accessible area in the Windows machine and I was able to run hadoop commands from SAS 9.3.

Unable to run Hadoop command

vivshri — Sun, 08 Feb 2015 14:15:11 GMT

Hi,

I am trying to setup hadoop connection with SAS 9.3.

filename cfg "C:/LocationOfMyHadoopConfiguration/hadoopconfig.cfg" ;

filename foo hadoop '/tmp/file1.txt' cfg=cfg user='hadoop' pass='mypassword';

data _null_;

file foo;

put "Hadoop Hadoop";

run;

ERROR: java.lang.ClassNotFoundException: com.dataflux.hadoop.DFConfiguration

ERROR: at java.net.URLClassLoader$1.run(Unknown Source)

ERROR: at java.security.AccessController.doPrivileged(Native Method)

ERROR: at java.net.URLClassLoader.findClass(Unknown Source)

ERROR: at java.lang.ClassLoader.loadClass(Unknown Source)

1. I have setup SAS_HADOOP_JAR_PATH in the environment variables so missing environment variable error is no longer coming.

2. I have downloaded all the hadoop jar files in SAS_HADOOP_JAR_PATH ( The jar files are for Hadoop 2.4.0 )

3. The picklist command "proc javainfo picklist 'hadoopbasics/hadoopwrapr.txt'; run;" showing

file:/C:/Program%20Files/SASHome/SASVersionedJarRepository/eclipse/plugins/dfx-hadoop-1.0_301000.

0.0.20121114190000_d3inhd31/dfx-hadoop-1.0.jar

file:/C:/Program%20Files/SASHome/SASVersionedJarRepository/eclipse/plugins/dfx-pig-1.0_301000.0.0

.20121114190000_d3inhd31/dfx-pig-1.0.jar

It seems like I have to specify required jar files in the picklist. is that correct? How to do that?

Appreciate any reply.