topic Re: Issues with special characters created with utf-8 encoding in SAS Programming

Issues with special characters created with utf-8 encoding

rdum96 — Thu, 11 Jan 2024 00:11:36 GMT

I'm importing a csv using the following code:

data finaldata;
  infile inputfile encoding='utf-8' truncover;
  input  mainline $varylen30000. ;
run;

However some of the characters like em dash, en dash etc are being converted to unusual characters such as below

I've had success gaining the hyphen of en dash using the code below

mainline=mainline(tranwrd,'96'x,'-')

but this is very specific to one of the cases I found. Is there someone way I could tackle all these special characters?

Re: Issues with special characters created with utf-8 encoding

Tom — Thu, 11 Jan 2024 01:18:31 GMT

If your SAS session is not also using UTF-8 encoding then it might not be possible to transcode every character in your UTF-8 text file into single byte encodings.

Also with single byte encodings what glyph is displayed for any particular byte will depend of the FONT you are using.

Re: Issues with special characters created with utf-8 encoding

Patrick — Thu, 11 Jan 2024 02:18:23 GMT

To test if this is a read/transcode or a write/display issue you could run below and see what gets printed.

data finaldata;
  em_dash='E28094'x;
  infile inp encoding='utf-8' truncover;
  input  mainline $10. ;
run;
proc print data=finaldata(obs=1);
run;

In my environment things are working:

Re: Issues with special characters created with utf-8 encoding

rdum96 — Thu, 11 Jan 2024 03:00:51 GMT

Ah gotcha! I believe I can't modify the SAS session 😞 I looked into the encoding and looks like it's 'latin1'. I was hoping the encoding option in the infile statement would help! I'll try to hard code everything for the time being!

Re: Issues with special characters created with utf-8 encoding

rdum96 — Thu, 11 Jan 2024 03:05:20 GMT

Thank you! I just checked and I get other special characters upon performing this 😞

Re: Issues with special characters created with utf-8 encoding

Tom — Thu, 11 Jan 2024 03:56:11 GMT

Read the file using ENCODING=ANY and then manually translate the non-ASCII codes into either single byte characters or some other things.

For example you could code like this to transcode the plus/minus symbol into the three character string +/- instead.

data want;
  infile 'textfile' encoding='any';
  input line $char100.;
  line = tranwrd(line,'C2B1'x,'+/-');
run;

Re: Issues with special characters created with utf-8 encoding

Patrick — Thu, 11 Jan 2024 06:07:26 GMT

For me in a single byte session with WLATIN1 the Hex value is 97.

filename inp "c:\temp\test_emdash_u8.csv";
data finaldata;
	em_dash='97'x;
	encoding="%sysfunc(getoption(encoding,keyexpand))";
	infile inp encoding='utf-8' truncover;
	input  mainline $10. ;
	hex=put(mainline,$hex20.);
run;
proc print data=finaldata(obs=1);
run;