ERROR: character with byte sequence 0x98 in encoding "WIN1251" has no equivalent in encoding "UTF8

[Gweeks]

Hello everyone, 

I've been tasked with registering a SAP table and pushing that table into Postgres within DI studio. I've registered the SAP table into SAS and once I try to create the table for Postgres it gives me the following error:

ERROR: CLI execute error: ERROR: character with byte sequence 0x98 in encoding "WIN1251" has no equivalent in encoding "UTF8"; Error while executing the query. I've also tried to use user-written code but it still gives out the same error. I'm still fairly new with SAS DI studio and writing tables into databases, I've included the preserve_col_name= option in the libname statement as well as the quote= option in the proc SQL statement. I noticed that the error might be coming from the column label, is there a way to get ride of that error. 

945        proc sql dquote=ansi
946        ;
947        create table pg_ecc."ausp"n as
948        select
949           ausp."MANDT"n length = 8
950              format = 3.
951              label = 'Client',
952           ausp."OBJEK"n length = 150
953              format = $150.
954              label = 'Key of object to be classified',
955           ausp."ATINN"n length = 10
956              format = $10.
957              label = 'Internal characteristic',
958           ausp."ATZHL"n length = 3
959              format = $3.
960              label = 'Characteristic value counter',
961           ausp."MAFID"n length = 3
962              format = $3.
963              label = 'Indicator: Object/Class',
964           ausp."KLART"n length = 9
965              format = $9.
966              label = 'Class Type',
967           ausp."ADZHL"n length = 4
968              format = $4.
969              label = 'Internal Counter for Archiving Objects by ECM',
970           ausp."ATWRT"n length = 90
971              format = $90.
972              label = 'Characteristic Value',
973           ausp."ATFLV"n length = 8
974              format = BEST12.
975              label = 'Internal floating point from',
976           ausp."ATAWE"n length = 9
977              format = $9.
978              label = 'Unit of Measurement',
979           ausp."ATFLB"n length = 8
980              format = BEST12.
981              label = 'Internal floating point value to',
982           ausp."ATAW1"n length = 9
983              format = $9.
984              label = 'Unit of Measurement',
985           ausp."ATCOD"n length = 3
986              format = $3.
987              label = 'Code for value dependency',
988           ausp."ATTLV"n length = 8
989              format = BEST12.
990              label = 'Tolerance from',
991           ausp."ATTLB"n length = 8
992              format = BEST12.
993              label = 'Tolerance to',
994           ausp."ATPRZ"n length = 3
37                                                         The SAS System                          Tuesday 16 March 2021 05:40:00 PM

995              format = $3.
996              label = 'Indicator: Tolerance Shown as a Percentage',
997           ausp."ATINC"n length = 8
998              format = BEST12.
999              label = 'Increment Within a Specified Interval',
1000          ausp."ATAUT"n length = 3
1001             format = $3.
1002             label = 'Classification: Author',
1003          ausp."AENNR"n length = 36
1004             format = $36.
1005             label = 'Change Number',
1006          ausp."DATUV"n length = 8
1007             format = DATE9.
1008             label = 'Valid-From Date',
1009          ausp."LKENZ"n length = 3
1010             format = $3.
1011             label = 'Deletion Indicator',
1012          ausp."ATIMB"n length = 10
1013             format = $10.
1014             label = 'Characteristic Number of User-Defined Data Type',
1015          ausp."ATZIS"n length = 3
1016             format = $3.
1017             label = 'Instance counter',
1018          ausp."ATSRT"n length = 4
1019             format = $4.
1020             label = 'Sort field for AUSP char. values',
1021          ausp."ATVGLART"n length = 3
1022             format = $3.
1023             label = 'Comparison Type for Char. Value: " " .. "7": Incl./Excl.',
1024          ausp."DATUB"n length = 8
1025             format = DATE9.
1026             label = 'Valid-to date'
1027       from
1028          work."ausp"n as ausp
1029       ;
NOTE: The DBIDIRECTEXEC option is not supported for this engine.
NOTE: SAS variable labels, formats, and lengths are not written to DBMS tables.
ERROR: CLI execute error: ERROR: character with byte sequence 0x98 in encoding "WIN1251" has no equivalent in encoding 
       "UTF8";
WARNING: File deletion failed for PG_ECC.AUSP.DATA.

NOTE: PROC SQL set option NOEXEC and will continue to check the syntax of statements.
1030       quit;
NOTE: The SAS System stopped processing this step because of errors.

I've included the proc SQL log details, your help will be greatly appreciated.

Gweek.

[Tom]

That seems strange as SAS knows how to translate that character to UTF-8.   perhaps there is some truncation of the translated string going on?

303   data _null_;
304    length x y $10 ;
305    x='98'x ;
306    y=kcvt(x,'WIN1251','UTF-8');
307    put (x y) (= :$quote.) ;
308    put (x y) (= :$hex4.) ;
309   run;

x="˜" y="˜"
x=9820 y=C298

[Gweeks]

@Tom  does the code you've shared show the truncated bits in the translated string and is there a way to go about fixing that error.

[Tom]

Check what encoding your SAS session is using.

It will probably work better to run SAS using UTF-8, but that might depend on your Postgres settings.

Note that labels are not going to be written into the foreign database so going to the trouble of trying to define them in your code is a waste of time for that destination.

You might try determining which observation and which variable is the one that is actually triggering the error.  Should not be that hard using a simple binary search strategy.  First eliminate half the variables and if that works then the variable is one in the other half.  Repeat until you figure out which variable.  Same for observations.  Try loading the first half of the observations if that works the problem is in something in the second half.