topic Re: Hadoop Data Library Connection Issue in SAS Data Management

Hadoop Data Library Connection Issue

mdavidson — Tue, 25 Mar 2014 20:21:26 GMT

Hello All --

I've created a new data library to Hadoop, the problem is that I'm not able to run any queries against this library. For example, here is the libname statement:

libname test hadoop subprotocol=hive2 port=10000 server="xxx.xx.com" schema=xxx;

When running the following query, I'm getting an error message

proc sql;

create table test as

select *

from

test.customer

where

state='RI';

quit;

ERROR: Unable to create stream service from /tmp/sasdata_16_01_48_861_00004. Use the debug option for more information is neither

an HDFS file nor an HDFS directory. A valid HDFS file or directory is required.

Doing some research it looks like most issues are due to JAR files not being present. I checked the JAR files listed in the SAS_HADOOP_JAR_PATH and here's what is listed -- everything seems to check out, any ideas on what might be causing the problem?

commons-cli-1.2.jar

core-site.xml

guava-11.0.2.jar

hadoop-auth-2.0.0-cdh4.5.0.jar

hadoop-common-2.0.0-cdh4.5.0.jar

hadoop-core-2.0.0-mr1-cdh4.5.0.jar

hadoop-hdfs-2.0.0-cdh4.5.0.jar

hive-exec-0.10.0-cdh4.5.0.jar

hive-jdbc-0.10.0-cdh4.5.0.jar

hive-metastore-0.10.0-cdh4.5.0.jar

hive-service-0.10.0-cdh4.5.0.jar

hive-site.xml

libfb303-0.9.0.jar

pig-0.11.0-cdh4.5.0.jar

protobuf-java-2.4.0a.jar

Re: Hadoop Data Library Connection Issue

mdavidson — Tue, 25 Mar 2014 20:33:39 GMT

Also here are the results of proc javainfo picklist 'hadoop/hdoopsasjars.txt'; run;

Picklist URLs:

file:/mnt/sas/sashome/SASVersionedJarRepository/eclipse/plugins/sas.hadoop.hivehelper_904000.0.0.20130522190000_v940/sas.hadoop.hivehelper.jar

file:/mnt/sas/sashome/SASVersionedJarRepository/eclipse/plugins/Log4J_1.2.15.0_SAS_20121211183158/log4j.jar

file:/mnt/sas/sashome/SASVersionedJarRepository/eclipse/plugins/commons_beanutils_1.8.2.0_SAS_20121211183319/commons-beanutils.jar

file:/mnt/sas/sashome/SASVersionedJarRepository/eclipse/plugins/commons_collections_3.2.1.0_SAS_20121211183225/commons-collections.jar

file:/mnt/sas/sashome/SASVersionedJarRepository/eclipse/plugins/commons_logging_1.1.1.0_SAS_20121211183202/commons-logging.jar

file:/mnt/sas/sashome/SASVersionedJarRepository/eclipse/plugins/jackson_1.9.7.0_SAS_20121211183158/jackson.jar

file:/mnt/sas/sashome/SASVersionedJarRepository/eclipse/plugins/slf4j_1.5.10.0_SAS_20121211183229/slf4j-api.jar

file:/mnt/sas/sashome/SASVersionedJarRepository/eclipse/plugins/slf4j_1.5.10.0_SAS_20121211183229/slf4j-log4j12.jar

Total URLs: 8

Re: Hadoop Data Library Connection Issue

JBailey — Thu, 03 Apr 2014 15:34:03 GMT

Hi,

Here is what I have for CDH 4.5.

avro-1.7.4.jar

commons-cli-1.2.jar

commons-collections-3.2.1.jar

commons-configuration-1.6.jar

commons-httpclient-3.1.jar

commons-logging-1.1.1.jar

guava-11.0.2.jar

hadoop-auth-2.0.0-cdh4.5.0.jar

hadoop-common-2.0.0-cdh4.5.0.jar

hadoop-core-2.0.0-mr1-cdh4.5.0.jar

hadoop-hdfs-2.0.0-cdh4.5.0.jar

hive-exec-0.10.0-cdh4.5.0.jar

hive-jdbc-0.10.0-cdh4.5.0.jar

hive-metastore-0.10.0-cdh4.5.0.jar

hive-service-0.10.0-cdh4.5.0.jar

libfb303-0.9.0.jar

log4j-1.2.17.jar

pig-0.11.0-cdh4.5.0-withouthadoop.jar

pig-0.11.0-cdh4.5.0.jar

protobuf-java-2.4.0a.jar

slf4j-api-1.6.1.jar

slf4j-log4j12-1.6.1.jar

Re: Hadoop Data Library Connection Issue

keds — Wed, 21 May 2014 09:58:22 GMT

Hi JBailey ,

we are trying to connect HortonWorks 2.2.0.2.0.6.0-101 and Cloudera CDH4.6 using SAS Access to Hadoop.

how to get the list of jars required ? does SAS Provides any documentation for this ?

Getting Below Error when trying to access.any thoughts about this ?

4 LIBNAME hdpcdh HADOOP PORT=10000 SERVER="XXXXXXXXXX" SCHEMA=default

15 USER=XXXXXXXXXXX PASSWORD=XXXXXXXXXX subprotocol=hive2 ;

ERROR: java.sql.SQLException: Could not establish connection to jdbc:hive2://XXXXXXXXXXXX:10000/default: java.net.ConnectException:

Connection timed out: connect

ERROR: Unable to connect to the Hive server.

ERROR: Error trying to establish connection.

ERROR: Error in the LIBNAME statement.

Thanks

keds

Re: Hadoop Data Library Connection Issue

DaveR_SAS — Wed, 21 May 2014 12:21:59 GMT

The Configuration Guide for SAS 9.4 Foundation includes a topic, "Configuring Hadoop JAR Files." Here is the PDF that includes that topic: http://support.sas.com/documentation/installcenter/en/ikfdtnunxcg/66380/PDF/default/config.pdf

Re: Hadoop Data Library Connection Issue

DaveR_SAS — Wed, 21 May 2014 12:56:47 GMT

Sorry: The doc link I supplied above is appropriate for SAS 9.4M1. I've just been told the following: "HW 2.2.0 and CDH4.6 came out after SAS 9.4M1 (which is the latest release in the field). I suspect their SAS consultant will have to help them find the correct files."

Re: Hadoop Data Library Connection Issue

JBailey — Wed, 21 May 2014 13:22:49 GMT

Hi Keds,

I don't have your exact setup, but I have something close. Keep in mind, that you cannot access HDP and CDH at the same time (from a single SAS process). This is because of the SAS_HADOOP_JAR_PATH= environment variable and the way JARs work.

Here is what I have for HDP 2.0:

commons-cli-1.2.jar

guava-12.0.1.jar

hadoop-auth-2.2.0.2.0.6.0-101.jar

hadoop-common-2.2.0.2.0.6.0-101.jar

hadoop-hdfs-2.2.0.2.0.6.0-101.jar

hadoop-mapreduce-client-common-2.2.0.2.0.6.0-101.jar

hadoop-mapreduce-client-core-2.2.0.2.0.6.0-101.jar

hadoop-mapreduce-client-jobclient-2.2.0.2.0.6.0-101.jar

hadoop-yarn-api-2.2.0.2.0.6.0-101.jar

hadoop-yarn-client-2.2.0.2.0.6.0-101.jar

hadoop-yarn-common-2.2.0.2.0.6.0-101.jar

hive-exec-0.12.0.2.0.6.1-101.jar

hive-jdbc-0.12.0.2.0.6.1-101.jar

hive-metastore-0.12.0.2.0.6.1-101.jar

hive-service-0.12.0.2.0.6.1-101.jar

httpclient-4.1.3.jar

httpcore-4.1.4.jar

libfb303-0.9.0.jar

pig-0.12.0.2.0.6.1-101.jar

protobuf-java-2.5.0.jar

Here is what I have for CDH 4.5:

avro-1.7.4.jar

commons-cli-1.2.jar

commons-collections-3.2.1.jar

commons-configuration-1.6.jar

commons-httpclient-3.1.jar

commons-logging-1.1.1.jar

guava-11.0.2.jar

hadoop-auth-2.0.0-cdh4.5.0.jar

hadoop-common-2.0.0-cdh4.5.0.jar

hadoop-core-2.0.0-mr1-cdh4.5.0.jar

hadoop-hdfs-2.0.0-cdh4.5.0.jar

hive-exec-0.10.0-cdh4.5.0.jar

hive-jdbc-0.10.0-cdh4.5.0.jar

hive-metastore-0.10.0-cdh4.5.0.jar

hive-service-0.10.0-cdh4.5.0.jar

libfb303-0.9.0.jar

log4j-1.2.17.jar

pig-0.11.0-cdh4.5.0-withouthadoop.jar

pig-0.11.0-cdh4.5.0.jar

protobuf-java-2.4.0a.jar

slf4j-api-1.6.1.jar

slf4j-log4j12-1.6.1.jar

Re: Hadoop Data Library Connection Issue

Nikolay — Thu, 29 May 2014 06:30:32 GMT

Hi Jeff,

Thank you for your list of JAR files for Hortonworks.

Re: Hadoop Data Library Connection Issue

JBailey — Mon, 09 Jun 2014 18:11:43 GMT

My pleasure. I hope you found it helpful. If you need anything, holler.

Re: Hadoop Data Library Connection Issue

pmohanty — Tue, 17 Jun 2014 05:58:20 GMT

HI,

We have Hortonworks 1.2 configured on our machine

The list of Hadoop Jar file we have set at SAS_HADOOP_JAR_PATH are:

guava-11.0.2.jar

hadoop-core-2.0.0-mr1-cdh4.3.1.jar

hive-exec-0.10.0-cdh4.3.1.jar

hive-jdbc-0.10.0-cdh4.3.1.jar

hive-metastore-0.10.0-cdh4.3.1.jar

hive-service-0.10.0-cdh4.3.1.jar

libfb303-0.9.0.jar

pig-0.11.0-cdh4.3.1.jar

protobuf-java-2.4.0a.jar

but other three jar files are not there with us:

hadoop-auth-2.0.0-cdh4.3.1.jar

hadoop-common-2.0.0-cdh4.3.1.jar

hadoop-hdfs-2.0.0-cdh4.3.1.jar

The list of Hadoop cluster configuration files that are set at:SAS_HADOOP_CONFIG_PATH are:

hdfs-site.xml

mapred-site.xml

core-site.xml

So my question is whether we need the other 3 jar files as mentioned above to Interface with Hadoop, connect to a specific Hadoop cluster and to To store data in HDFS using the SPD Engine.

As you have mentioned earlier that we dont need SAS/ACCESS for To Interface with hadoop cluster using SPD Engine.

But in my case when we just ran the below libname statement:

LIBNAME hdplib SPDE '/SAS/sangram/data' HDFSHOST=DEFAULT;

we are getting some issue regarding Can't get Kerberos configuration.

So,Please help me regarding this and also do we need the other 3 jars files??? and if required can you please provide them.It will really be grateful.

Thank You and Waiting for a Reply.

Sangramjit

Re: Hadoop Data Library Connection Issue

JBailey — Wed, 18 Jun 2014 13:08:54 GMT

Hi Sangramjit,

You may want to open a different discussion for this topic. The SPDE interface is distinct from SAS/ACCESS. I notice that you say you are using Hortonworks, but you are listing Cloudera JAR files. This will lead to problems. You want to make certain that your JAR files match your distribution.

Troubleshooting Kerberos issues can be fraught with peril. In order to do that someone needs to go through your XML files. If your problem is Kerberos, it is best to open a track with SAS Technical Support.

Best wishes,

Jeff

Re: Hadoop Data Library Connection Issue

pmohanty — Thu, 19 Jun 2014 06:21:14 GMT

Hi Jeff,

I have a list of Hadoop Jar files for Hortonworks 1.2 and placed on my machine:

protobuf-java-2.4.1.jar

libfb303-0.9.0.jar

hive-service-0.11.0.1.3.3.4-2.jar

hive-metastore-0.11.0.1.3.3.4-2.jar

hive-jdbc-0.11.0.1.3.3.4-2.jar

hive-exec-0.11.0.1.3.3.4-2.jar

hadoop-core-1.2.0.1.3.3.4-2.jar

guava-11.0.2.jar

pig-0.11.1.1.3.3.4-2.jar

But the other jar files for Hortonworks 1.2 such as:

hadoop-auth-jar

hadoop-common-.jar

hadoop-hdfs-.jar

are not there with us.Rest all configuration files are present and they are also set by the respective Environment variables.

While executing the sas script we are getting the following issue:

ERROR: Could not connect to HDFS.

ERROR: Libref HDPLIB is not assigned.

ERROR: Error in the LIBNAME statement.

ERROR: Call to method org.apache.hadoop.fs.FileSystem::get(URI, Configuration) failed.

ERROR: java.lang.ExceptionInInitializerError

java.lang.IllegalArgumentException: Can't get Kerberos configuration

So,can you please tell me whether we are getting this type of error for missing of above 3 jar files or some other issue is there.

Thanks,

Sangramjit

Re: Hadoop Data Library Connection Issue

pmohanty — Thu, 19 Jun 2014 06:24:23 GMT

Jeff,can you go through the Disscussion: SPDEngine:Storing Data in the Hadoop Distributed File System opened by Sangramjit Panda

Thanks,

Sangramjit

Re: Hadoop Data Library Connection Issue

AnnaBrown — Thu, 19 Jun 2014 14:51:35 GMT

Here is the link to the above-mentioned thread opened by Sangramjit.

SPDEngine:Storing Data in the Hadoop Distributed File System

Re: Hadoop Data Library Connection Issue

fatcat — Tue, 27 Oct 2015 21:04:10 GMT

Hi, i am trying to connect to Hortonworks with SAS/Access for Hadoop. Able to get the klist, and able to ssh from the SAS Client machine to the one of the nodes. But getting an error when trying to connect via SAS EG. Any idea?

libname zhdplib hadoop subprotocol='hive2' server='myserver' schema=myschema; user='user123' pwd='Test123';

ERROR: Unable to connect to the Hive server.
ERROR: Error trying to establish connection.
ERROR: Error in the LIBNAME statement.

Alex

Re: Hadoop Data Library Connection Issue

JBailey — Wed, 28 Oct 2015 17:28:07 GMT

Hi Fatcat,

Do you mind starting a new thread for your issue?

Best wishes,
Jeff

Re: Hadoop Data Library Connection Issue

fatcat — Wed, 28 Oct 2015 18:18:39 GMT

done.

Re: Hadoop Data Library Connection Issue

JBailey — Wed, 28 Oct 2015 18:20:14 GMT

Thanks!