topic Re: redirecting HDFS TEMP FILES in SAS Data Management

redirecting HDFS TEMP FILES

vorkady — Thu, 06 Sep 2018 14:34:55 GMT

We are encountering issue in redirecting HDFS temp Directory using option hdfs_tempdir in the libname syntax. My id has full privilege on the temp directory path but getting access error , which is misleading. Please advise if any community members encountered similar issues and workaround used to resolve the issue. I have included below code and SAS logs for your ready reference. Appreciate if you could help.

libname x3 hadoop AUTHDOMAIN=&auth_domain. server="xxxxt"
uri="jdbc:hive2://xxxx3/;ssl=true;transportMode=http;httpPath=gateway/default/hive2" schema="sbx_cr_mdl__user_xx"
DBMAX_TEXT=255 READ_METHOD=HDFS hdfs_tempdir="/hdp-staging/home/xxx";

Logs:

! hdfs_tempdir="/hdp-staging/home/svorkad";
ERROR: HDFS_TEMPDIR directory /hdp-staging/home/xxx does
5 The SAS System
14:31 Wednesday, September 5, 2018

not exist or is inaccessible.
ERROR: Error in the LIBNAME statement.
35

Re: redirecting HDFS TEMP FILES

JBailey — Fri, 07 Sep 2018 18:14:47 GMT

Hi @vorkady,

If Kerberos is being used I would bet that your Kerberos ticket will not allow writing to that location.

Try writing a file to the HDFS_TEMPDIR= location using PROC HADOOP.

PROC HADOOP displays better (more descriptive) error messages because it goes through fewer Hadoop layers when writing the file.

There may be an example in my SGF workshop materials.

https://github.com/Jeff-Bailey/SGF2016_SAS3880_Insiders_Guide_Hadoop_HOW

If you have something like HUE available it is worth a try too.

Best wishes,
Jeff

Re: redirecting HDFS TEMP FILES

vorkady — Fri, 07 Sep 2018 18:24:31 GMT

Thanks Jeff,
We also encountering issues displaying migrated Netezza table data in EG , which is visible in non sas applications such as Aginity hadoop work bench but not in EG . Further it is observed that data in table is visible in EG if column attributes set to string. I did open track with SAS support . Appreciate if you could help.

Re: redirecting HDFS TEMP FILES

vorkady — Thu, 13 Sep 2018 17:22:02 GMT

Hi,

I am able to resolve the issue by uploading the required hive site xml file . We are now able to redirect temp direct in libname and also able to view table data in EG.

Note from Jeff and other contributors immensely helped us to resolve our Hadoop SAS config issues.