VA server and uploading to Hadoop with SAS/ACCESS

[dkassis]

Hi,

I have a program that pulls data from Hadoop, performs several transformations, merges, etc. in the SAS work directory, then I want to load that dataset into VA and back into Hadoop.  Our VA license gives us the ability to run proc sql against Hadoop via SAS/ACCESS and the SASAppVa server.  However I get the following error when I try to load the data back into Hadoop from the work directory.

NOTE: SAS variable labels, formats, and lengths are not written to DBMS tables.

ERROR: org.apache.hadoop.security.AccessControlException: SIMPLE authentication is not enabled. Available:[TOKEN, KERBEROS]

NOTE: Validate the contents of the Hadoop configuration file and ensure user permissions are correct.

ERROR: Unable to create HDFS file: /tmp/sasdata-2019-08-09-08-34-15-931-e-00001.dlv. Possible causes are failure to connect to the

Hadoop HDFS service or user 'svc_sas_hdp_prod' lacks write permission to HDFS directory /tmp.

NOTE: The DATA step has been abnormally terminated.

NOTE: The SAS System stopped processing this step because of errors.

When I run this on the OA Server I do not get an error.  This looks like a permissions item, but is there any limitations I need to be aware of when writing from the work directory to Hadoop using the VA server? 

The reason I am going this route is I want to schedule this project so it automatically runs everyday and updates the reports I have in VA as well as create a table in Hadoop that can be queried against.  The way we are doing it now is to run on OA, then switch servers and grab the table from Hadoop and load into VA using the data 'Upload to LASR' function.  I want to be able to get rid of that manual step and automate the whole process.

Thanks,

Dan 

[Anand_V]

Hi @dkassis,

I can see two error's you have listed in your post.

1) ERROR: org.apache.hadoop.security.AccessControlException: SIMPLE authentication is not enabled. Available:[TOKEN, KERBEROS]

My understanding of the above error is that, your hadoop cluster is enabled for only Token, KERBEROS Authentication types. you will have to either provide Auth token or krb ticket to connect. If you don't have this info, you can check with your hadoop admin on how to get this.

2) ERROR: Unable to create HDFS file: /tmp/sasdata-2019-08-09-08-34-15-931-e-00001.dlv. Possible causes are failure to connect to the Hadoop HDFS service or user 'svc_sas_hdp_prod' lacks write permission to HDFS directory /tmp.

SAS when process data to write back to hive, it creates temporary files in /tmp of HDFS cluster. User which would get connected to Hadoop cluster should have read/write permissions on this path in HDFS. Usually for Cloudera Hadoop distributions this path permissions is controlled by sticky bit and user doesn't have permissions to write. you can ask your hadoop admin to grant permission to your user. If they don't allow this, you can write your tmp files into another path in HDFS where your ID has access. You can use HDFS_TEMPDIR= option in your libname statement to provide alternate tmp path.

HDFS_TEMPDIR='path'

specifies the path to the HDFS directory where SAS reads and writes temporary data.

Default: HDFS_TEMPDIR='/tmp'

https://documentation.sas.com/?docsetId=acreldb&docsetTarget=n1h398otek0j00n1itib5k7ch738.htm&docset...

HTH

Thanks!