topic Re: SAS Error writing to Hadoop in SAS Data Management

SAS Error writing to Hadoop

Lcopello — Fri, 14 Jul 2017 20:04:10 GMT

Hi All,

When trying to write a sas dataset out to Hadoop I get the error below. I will let everyone know that I have tried many of the fixes that have been posted but nothing has worked. In our data lake it looks like it is creating the table but it creates is the headers. We were able to write to the data lake 2 weeks ago but we have switched to a new cluster and we can no longer write from SAS EG, SAS EM or SAS Studio. Any help would be appreciated. The code that I use is also below.

ERROR: Execute error on statement: LOAD DATA INPATH '/tmp/sasdata-2017-07-14-14-02-27-980-e-00008.dlv' OVERWRITE INTO TABLE

sastmp_07_14_14_02_28_143_00009. Could not load /tmp/sasdata-2017-07-14-14-02-27-980-e-00008.dlv into table

sastmp_07_14_14_02_28_143_00009 in schema UTILITY. A common cause of this issue is conflicting HDFS permissions between the

data file and the Hive warehouse directory for the table. Another possible cause is the "sticky" bit set on HDFS directory

/tmp.

______________________________________________________________________________________

Code used that gets the error

--------------------------------------------------------------------------------------

options mprint mlogic symbolgen mlogicnest compress=yes;
options set=SAS_HADOOP_CONFIG_PATH='D:\\hadoopcfg';
options set=SAS_HADOOP_JAR_PATH='D:\\hdp22';
OPTIONS SET=SAS_HADOOP_RESTFUL='1';

libname liz hadoop server="xxxxx" PORT=10000 database=user_xxxxx SUBPROTOCOL=hive2
user="xxxxx" password="xxxxxxxx" HDFS_TEMPDIR="/tmp" ;

proc sql ;
drop table utility.HolidayCalendar2;

create table utility.HolidayCalendar2 (DBCREATE_TABLE_OPTS='STORED AS ORC') as select * from utility.referenceCalendar;

quit;

Re: SAS Error writing to Hadoop

Patrick — Sun, 16 Jul 2017 14:08:10 GMT

@Lcopello

Well, if the environment changed then I'd take the information in the SAS Log very seriously:

A common cause of this issue is conflicting HDFS permissions between the

data file and the Hive warehouse directory for the table. Another possible cause is the "sticky" bit set on HDFS directory

Have you already excluded the above two possible causes?

If you can't get to the bottom of this then I suggest you contact SAS TechSupport as you will likely have to provide site specific information which you can't post into a public forum.

Re: SAS Error writing to Hadoop

Lcopello — Mon, 17 Jul 2017 12:14:15 GMT

Patrick,

We have tried the things that were mentioned in the error plus some other things that we had found online while researching hte issue.

Thanks for your help.

Re: SAS Error writing to Hadoop

Lcopello — Thu, 20 Jul 2017 12:26:38 GMT

We found a solution to the problem. We ended up having to change the SAS config file for HADOOP.

Thanks,

Re: SAS Error writing to Hadoop

Patrick — Thu, 20 Jul 2017 14:19:30 GMT

@Lcopello

Can I suggest you describe the issue you had and the resolution to the issue as detailed as possible and then mark this as the solution.

This could help others with similar challenges to use your resolution as guidance.