topic Re: Problem importing csv file with UTF-8 encoding in SAS Procedures

Problem importing csv file with UTF-8 encoding

Paolo_Corti — Tue, 10 Aug 2021 12:50:07 GMT

Hallo,

I am trying to import a csv file. In this file the first line, the one containing the variables name, contains some names with utf-8 characters.

I'm working with SAS 9.4.

First try to import the code without considering the special characters using the "wlatin1" encoding.

filename P1 "C:\Temp\Test_UTF_8.csv" encoding='wlatin1';

proc import datafile=P1 dbms=csv replace out=raw;
delimiter=';';
run;

In this case the code is working but the columns containing the special characters like "ä".

When I try to change the encoding using "utf-8"

filename P1 "C:\Temp\Test_UTF_8.csv" encoding='utf-8';

proc import datafile=P1 dbms=csv replace out=raw;
delimiter=';';
run;

the code is not working and I get the error:

ERROR: Invalid string.
FATAL: Unrecoverable I/O error detected in the execution of the DATA step program.
Aborted during the INITIALIZE EXECUTION phase.

Has someone a solution to this problem or can give me an hint where to look.

Thanks a lot

Paolo

In attachment you find the file I'm trying to import and the log file.

Re: Problem importing csv file with UTF-8 encoding

Ksharp — Tue, 10 Aug 2021 13:32:01 GMT

You need a special option " options validvarname=any ".

options validvarname=any;
filename P1 "C:\Temp\Test_UTF_8.csv" encoding='wlatin1';
proc import datafile=P1 dbms=csv replace out=raw;
delimiter=';';
run;

Re: Problem importing csv file with UTF-8 encoding

Tom — Tue, 10 Aug 2021 13:39:34 GMT

Your file does not contain UTF-8 characters In fact the extended ASCII characters it does have are using some of the values that UTF-8 uses to indicate multi-byte character encoding and that is what is causing the errors when you treat it as a UTF-8 encoded text.

Note since you are reading a text file you can just write your own data step to read the data. Then you can control the variable names that will be used.

Re: Problem importing csv file with UTF-8 encoding

Paolo_Corti — Tue, 10 Aug 2021 13:46:10 GMT

Thank a lot for the answers!