Desktop productivity for business analysts and programmers

Encoding Issue UTF8 to UTF 16-le

Accepted Solution Solved
Reply
New Contributor
Posts: 4
Accepted Solution

Encoding Issue UTF8 to UTF 16-le

Can someone help me with issue im facing ?

 

I'm woring with dataset from CRM  containing contacts. Names, surnames . dataset encoding is UTF-8, my sessions encoding is UTF-8. so up to my knowledge everything should work. But it's not. I'm recievieng such errror message (attached):

 

I need to remove diacritics etc.. so I'm using translate function to remove diacritics (Slovak)

%let TRANSLATE_FROM='ÁÄČĎÉÍĹĽŇÓÔŔŠŤÚÝŽ';
%let TRANSLATE_TO='AACDEILLNOORSTUYZ';

 

I'm also having list of polish letters I want to replace.. the thing is I can't

 

sas ,I 'dont know why, is trying to transcode data from utf-8 to utf 16-le ... 

 

i tried to parse string to find out what type of letters are present in string. Worked only with A='Tomas'

 

data A;
/*A = 'АлександраЪъ';*/
/*A= "Zażółć gęślą jaźń";*/
A='Tomas';
D=length(trim(a));
array parsed_vars(40) $ new_var1-new_var40;
Do i = 1 to D;
parsed_vars(i)=substr(a,i,1);
end;
run;

 

 

I'm running desktop version of SAS EG 9.4.

 

 


sas_error_encoding.jpg

Accepted Solutions
Solution
‎08-02-2017 06:42 AM
SAS Super FREQ
Posts: 326

Re: Encoding Issue UTF8 to UTF 16-le

Have you run similar code in EG to confirm the session encoding?:

 

proc options group=languagecontrol;

run;

 

If you have confirmed it is UTF-8, I suspect you are running against your Local SAS server, rather than a remote SAS server?  Unlike the SAS Bridge protocol (remote server communication) which transmits in UTF-8, in COM (Local), strings have to be transmitted in UNICODE, UTF-16.  In this case, if there are characters that can't be transcoded from UTF-8 to UTF-16, you'll get the transcoding error.

View solution in original post


All Replies
Super User
Posts: 10,041

Re: Encoding Issue UTF8 to UTF 16-le

It is famous error for EG. You have unprint or illegal character for EG, it is not problem about encoding.

Here are two option.

1) clean your sas table. to make all the character are legal for EG.

2) export this table into a CSV or TXT file ,then import it back by writing some sas code.

 

 

New Contributor
Posts: 4

Re: Encoding Issue UTF8 to UTF 16-le

What are illegal characters for EG? Do you mean reserved one like *&%$;... etc ? 

Solution
‎08-02-2017 06:42 AM
SAS Super FREQ
Posts: 326

Re: Encoding Issue UTF8 to UTF 16-le

Have you run similar code in EG to confirm the session encoding?:

 

proc options group=languagecontrol;

run;

 

If you have confirmed it is UTF-8, I suspect you are running against your Local SAS server, rather than a remote SAS server?  Unlike the SAS Bridge protocol (remote server communication) which transmits in UTF-8, in COM (Local), strings have to be transmitted in UNICODE, UTF-16.  In this case, if there are characters that can't be transcoded from UTF-8 to UTF-16, you'll get the transcoding error.

New Contributor
Posts: 4

Re: Encoding Issue UTF8 to UTF 16-le

Posted in reply to CaseySmith

please see log attached, Im running UTF8 on local machine (laptop)

 

Im working with data downloaded from PostrgeSQL DB. Customers can create their profile from web or they are imported from our internal systems. Is there way how to detect characters which can't be translated and remove them i.e. using compress function?

 

I don't have full IT background as I studied economics so this is beyond my knowledge 

 

Acording this paper http://unicode.org/faq/utf_bom.html  I suppose that UTF16 covers entire UTF8 and adds somthing more.. 

New Contributor
Posts: 4

Re: Encoding Issue UTF8 to UTF 16-le

Posted in reply to CaseySmith

i simply didn't use correct string functions able to work with double-byte character expressions (UTF8) 

 

http://morgan.dartmouth.edu/Docs/sas92/support.sas.com/documentation/cdl/en/nlsref/61893/HTML/defaul...

 

SAS could put such error message to log - use of inapropriate function

☑ This topic is solved.

Need further help from the community? Please ask a new question.

Discussion stats
  • 5 replies
  • 370 views
  • 0 likes
  • 3 in conversation