SAS Access to connect with Hadoop HDFS Engine

[Balarama1]

Hi SAS Friends,

Version=9.4

Platform_long=Linux for X64
License_prod=SAS

We are connecting SAS to Hadoop with HDFS Engine from cloudera Manager and ServiceDiscoveryMode is zooKeeper and we used kerberos secuity authentication(without username and password)

We have not found any libname for connecting HDFS, but by using Proc Hadoop we are connected to WebHDFS.

Issue:By using Proc Hadoop we need to read the data from HDFS Server, but to read HDFS commands are not there.

By using copy to local command we are downloading the data from HDFS Server, from this Process we are facing Server Storage utilization issue.

Summary is that we want to read data directly from HDFS Server, instead of downloading data from HDFS to SAS Server. 

Suggest us if it possible with Libname for connecting to HDFS Server from cloudera Manager and ServiceDiscoveryMode is zooKeeper and kerberos secuity authentication

 Please suggest and support us to resolve. 

[AhmedAl_Attar]

Hi @Balarama1 

check out the following on-line doc  SAS Help Center: SAS 9.4 Hadoop Configuration Guide for Base SAS and SAS/ACCESS, Fifth Edition and see if you are missing anything!?

Once you ensured your configuration is set correctly, you can checkout this on-line doc SAS Help Center: SAS/ACCESS Interface to Hadoop for information about setting up Hadoop Libname statement.

Hope this helps

[Balarama1]

Hi @AhmedAl_Attar ,

Thank you for response.

From the following links we already done the pre-requisite and by using Proc Hadoop for HDFS we are able to connected to HDFS Server. PF attached doc.

And you shared the link for Libnames but in our process we don't have username and password by using kerberos authetication we are working and to mention the Hadoop server and port we are processing through Zookeper ServiceDiscoveryMode. And we observe that in libname statement link it is showing for Hive Server connection but here we are connecting to HDFS Server.

Please support to resolve this.

[Balarama1]

Hi @AhmedAl_Attar ,

From the below libname we have tried to connect to HDFS.

libname HD Hadoop server='masternode' port=9870(webHDFS port);HDFS_PRINCIPAL="hdfs/_HOST@CDPDEVMASTER01";

From the attached zip file you can see the screenshot, we use that libname for connecting to HDFS with kerberos security authentication..

 And find the HDFS error log file shared in zip file.

We added in server as name node with Web HDFS Port

Please suggest us to resolve this. 

[AhmedAl_Attar]

@Balarama1 

I would highly recommend you open a SAS Tech Support track for this issue. 

SAS Tech support come across all kinds of deployments and configurations, plus they have direct line to R&D staff. 

[LinusH]

Try to avoid attaching zip archives and similar since many avoid to download this kind of content.

Logs and similar can perfectly be pasted into the post itself.