Integration Job Failing Due to non English Characters:Encoding UTF8

[samstheway79]

Hi All

I am reading data from a HADOOP table on a simple DI job. One of the columns in the source object has some values that are non English character set, e.g BANČNA PODRUŽNICA.

The job fails with the following message:

ERROR: Data from column 'fincl_acnt_num' in row 273210 of the result set was not presented in Hadoop UTF-8 format. The length of this data is 22 bytes, and the first 7 characters are '2025088'. Adding -JREOPTIONS (-Dfile.encoding=UTF-8) to the SAS invocation may circumvent the issue. Otherwise the data should be corrected to UTF-8 format.

I have gone through NLS forums but none of the solutions apply or I wasn't able to apply for my case on DI Studio.

Thanks!

Samir

[Rama_V]

I have come across similar situation. It might be case that your SAS Server is not configured to handle UTF-8 encoding data.

https://blogs.sas.com/content/sgf/2017/05/19/demystifying-and-resolving-common-transcoding-problems/

Check you session encoding

http://support.sas.com/documentation/cdl/en/nlsref/63072/HTML/default/viewer.htm#p12mamjt5ccmshn1idw...

Easy option is to convert those words or remove those characters using Base SAS running under Unicode session.

http://support.sas.com/kb/15/597.html

Below one might be good read for you and there is conversion code on Page 7

http://support.sas.com/resources/papers/Multilingual_Computing_with_SAS_94.pdf

From my experience converting server config you need have admin access, there will be additional remediation steps required based on your data storage. 

http://support.sas.com/kb/51/586.html

Best Wishes,

Rama