SAS Studio to hadoop UTF-8 Error

[JanVanHeyden]

Hi,

I'm using the libname to assign a schema on hadoop. I already tested writing data via SAS Studio to hadoop and it works fine except for one table called "table1" with a string column called 'content'.

If I try the following:

libname myhive 	hadoop subprotocol=hive2 
				port=10000 
				host= "myhost" 
				schema= "default"
				; 

data myhive.test;
	merge myhive.table1 (in=a)
		   myhive.table2 (in=b)
		;
	by id;
	if a*b;
run;

I get the following error log:

 ERROR: Data from column 'content' in row 1435 of the result set is not in Hadoop UTF-8 format. The length of this data is 86 bytes, 
        and the first 13 characters are 'home/Finanzen'. The data should be corrected to UTF-8 format.

I tested which session encoding my SAS Studio uses via:

proc options option=encoding;    
run;

and it says:

     SAS (r) Proprietary Software Release 9.4  TS1M3
 
  ENCODING=LATIN1   Specifies the default character-set encoding for the SAS session.

Maybe this triggers the error above. I found, however, no way to change my session encoding.

[JanVanHeyden]

Hi,

I tested some more. Even if I try to write on my work libary I get an Error

Job:

data work.test;
	set myhive.table1;
run;

Here I get a little bit more Information for the Error:

 HADOOP_70: Prepared: on connection 6
 SELECT * FROM `table1`
  
 59         run;
 
 INFO: Data file WORK.TEST.DATA is in a format that is native to another host, or the file encoding does not match the 
 session encoding. Cross Environment Data Access will be used, which might require additional CPU resources and might reduce 
 performance.
 ERROR: Data from column 'content' in row 297906 of the result set is not in Hadoop UTF-8 format. The length of this data is 94 
        bytes, and the first 12 characters are 'inteligentne'. The data should be corrected to UTF-8 format.
 NOTE: The DATA step has been abnormally terminated.
 NOTE: The SAS System stopped processing this step because of errors.

[ApoorvM]

Did you get solution for the same, or anybody can help with this issue.

This is really critical, and I assume many people facing the similar sort of issue.

Any one from SAS company Please contribute to this.

[JanVanHeyden · Accepted solution]

Hi,

I finally found a solution that works for me. Basically the problem seems to be that within hadoop everything is formated in utf-8 while most SAS Studio Sessions run in latin1 (or something else country specific). While I found no way of changing the SAS Studio Session there is a command that suppresses these kind of errors and still writes the data from hadoop to temporary SAS Files:

According to my example above the change to the code would be to add behind the hive table this command (transcode_fail=silent):

data work.test;
	set myhive.table1 (transcode_fail=silent);
run;

[ApoorvM]

Thank you so much JanVanHeyden !!

This is really helpful, I tried the same and it works for me as well!

Thanks again!!

ApoorvM