Abnormal characters in sas file

[Eileen1496]

Hi all,

I got a 4gb SAS dataset with company information. The company name column is very important to me. However, I found though most of the observations are in normal format, e.g. "Birly Co", there are many rows has values like "ÃÂ". Do you know if this is a problem in the original dataset that I cannot solve as a data user, or it is a problem of encoding on my computer?  

I actually tried encoding it to utf-8 or utf-16le, utf-16be in python, but still I got such weird characters.

Thank you!

[Kurt_Bremser]

What is your session encoding?

[Eileen1496]

Hi Kurt, what do you mean by "session encoding"? I tried utf-8, utf-16le,
utf-16be in python. In sas I just open the file directly because it is
originally in sas7bdat format. And it also shows the abnormal chars.

[Kurt_Bremser]

In SAS, run

proc options option=encoding value;
run;

to see how the session encoding is set.

[Eileen1496]

Hi Kurt! It return the following to me:

SAS (r) Proprietary Software Release 9.4 TS1M6

Option Value Information For SAS Option ENCODING
Value: WLATIN1
Scope: SAS Session
How option value set: Config File
Config file name:
C:\Program Files\SASHome\SASFoundation\9.4\nls\en\sasv9.cfg

Should I change anything so this problem can be fixed?

[Kurt_Bremser · Accepted solution]

You will need to have a session started with utf-8 encoding to properly use the dataset. Talk with your SAS administrators how to get one.

[Eileen1496]

Dear Kurt, thanks for pointing out the problem! I would never realize that alone probably, I will talk with them.

[Eileen1496]

Dear Kurt, thanks for pointing out the problem! I would never realize that alone probably, I will talk with them.

[GPatel]

You can try :

Suppose your abnormal characters in SAS file is in "Comments" column/field. 

Use code below to remove/replace weird/abnormal characters.

Comments=KCVT(compress(Comments,,'kw'), 'wlatin1', 'UTF-8');
or

Comments =compress(Comments, , 'kw');