Encoding Issue UTF8 to UTF 16-le

[mouti]

Can someone help me with issue im facing ?

I'm woring with dataset from CRM  containing contacts. Names, surnames . dataset encoding is UTF-8, my sessions encoding is UTF-8. so up to my knowledge everything should work. But it's not. I'm recievieng such errror message (attached):

I need to remove diacritics etc.. so I'm using translate function to remove diacritics (Slovak)

%let TRANSLATE_FROM='ÁÄČĎÉÍĹĽŇÓÔŔŠŤÚÝŽ';
%let TRANSLATE_TO='AACDEILLNOORSTUYZ';

I'm also having list of polish letters I want to replace.. the thing is I can't

sas ,I 'dont know why, is trying to transcode data from utf-8 to utf 16-le ... 

i tried to parse string to find out what type of letters are present in string. Worked only with A='Tomas'

data A;
/*A = 'АлександраЪъ';*/
/*A= "Zażółć gęślą jaźń";*/
A='Tomas';
D=length(trim(a));
array parsed_vars(40) $ new_var1-new_var40;
Do i = 1 to D;
parsed_vars(i)=substr(a,i,1);
end;
run;

I'm running desktop version of SAS EG 9.4.

[Ksharp]

It is famous error for EG. You have unprint or illegal character for EG, it is not problem about encoding.

Here are two option.

1) clean your sas table. to make all the character are legal for EG.

2) export this table into a CSV or TXT file ,then import it back by writing some sas code.

[mouti]

What are illegal characters for EG? Do you mean reserved one like *&%$;... etc ? 

[CaseySmith · Accepted solution]

Have you run similar code in EG to confirm the session encoding?:

proc options group=languagecontrol;

run;

If you have confirmed it is UTF-8, I suspect you are running against your Local SAS server, rather than a remote SAS server?  Unlike the SAS Bridge protocol (remote server communication) which transmits in UTF-8, in COM (Local), strings have to be transmitted in UNICODE, UTF-16.  In this case, if there are characters that can't be transcoded from UTF-8 to UTF-16, you'll get the transcoding error.

[mouti]

please see log attached, Im running UTF8 on local machine (laptop)

Im working with data downloaded from PostrgeSQL DB. Customers can create their profile from web or they are imported from our internal systems. Is there way how to detect characters which can't be translated and remove them i.e. using compress function?

I don't have full IT background as I studied economics so this is beyond my knowledge 

Acording this paper http://unicode.org/faq/utf_bom.html  I suppose that UTF16 covers entire UTF8 and adds somthing more.. 

[mouti]

i simply didn't use correct string functions able to work with double-byte character expressions (UTF8) 

http://morgan.dartmouth.edu/Docs/sas92/support.sas.com/documentation/cdl/en/nlsref/61893/HTML/defaul...

SAS could put such error message to log - use of inapropriate function