topic Re: Solution for ERROR: Some character data was lost during transcoding in the dataset in SAS Programming

Solution for ERROR: Some character data was lost during transcoding in the dataset

lokendra_devangan_corecompete_com — Fri, 10 Jun 2016 17:04:38 GMT

When I search for the mentioned error I get the link solution provided adminstration oriented but developers do not have access.

http://support.sas.com/kb/52/716.html

Please see my solution below

proc options option=config; run;

proc options group=languagecontrol; run;

/* Show the encoding value for the problematic data set */

%let dsn=item_information_may16;

%let dsid=%sysfunc(open(&dsn,i));

%put &dsn ENCODING is: %sysfunc(attrc(&dsid,encoding));

/*Renaming item desc file (encoding=any) allowed reading****************************/

data tmp.item_info_curr;

set hc.item_information_may16 (encoding=any);

run;

/*gave error*/

data item_info_curr;

set item_information_may16;

run;

Re: Solution for ERROR: Some character data was lost during transcoding in the dataset

Ksharp — Sat, 11 Jun 2016 03:16:46 GMT


You could try this code to avoid these ERROR.


libname xx cvp '/folders/myfolders/';
proc copy in=xx out=work noclone;
 select class;
run;

Re: Solution for ERROR: Some character data was lost during transcoding in the dataset

ehbales — Mon, 09 Jan 2017 20:46:44 GMT

The cause of the error may be truncation rather than transcoding. If your SAS session encoding is UTF-8 and your data set encoding is another encoding, the error may be telling you that the length of one or more character variables is not long enough to hold the UTF-8 version of a string. Some characters require more bytes in UTF-8 than they did in the "legacy" encodings that SAS supports.

If that is the scenario you have, the CVP engine may be helpful to you. Several resources are available that you may find useful:

The National Language Reference Guide for 9.4 has some sample code showing how to use CVP. See the section "Avoiding Character Data Truncation by Using the CVP Engine" in the "Transcoding in NLS" chapter of the NLS Concepts section.

The white paper titled Multilingual Computing with SAS 9.4 also discusses CVP and other issues related to working with multilingual data. The paper was written for 9.4, but many of the features documented there are available with earlier releases of SAS 9.

http://support.sas.com/resources/papers/Multilingual_Computing_with_SAS_94.pdf

Re: Solution for ERROR: Some character data was lost during transcoding in the dataset

india2016 — Thu, 30 Mar 2017 13:49:35 GMT

@Ksharp

I tried below code but still, I got the same error

"Some character data was lost during transcoding in the dataset XX.AA. Either the data
contains characters that are not representable in the new encoding or truncation
occurred during transcoding."

libname xx cvp '/folders/myfolders/';
proc copy in=xx out=work noclone;
 select class;
run;

what should I do to read Hindi Language words in SAS 9.4 (English with DBCS)?

Re: Solution for ERROR: Some character data was lost during transcoding in the dataset

ehbales — Fri, 11 Aug 2017 16:03:24 GMT

By default, the configuration file for the configuration called "English with DBCS Support" sets the ENCODING (SAS session encoding) to an encoding that supports Japanese. The Hindi characters are not supported by the Japanese encoding.

If you use the configuration that is created under the "nls/u8" directory, that will set the SAS session encoding to UTF-8. The Hindi characters are supported there.

If you are already using UTF-8, then try increasing the multiplier of the CVP engine. Hindi characters require 3 bytes per character. You probably need to set the CVPMULTIPLIER= option to 3.

For more information, check this comprehensive article:

Demystifying and resolving common transcoding problems

You'll learn about:

The CVP engine to help convert data from one encoding to another
ENCODING=ASCIIANY and OUTENCODING=ASCIIANY options for suppressing transcoding

Re: Solution for ERROR: Some character data was lost during transcoding in the dataset

Tom — Thu, 30 Mar 2017 14:46:53 GMT

The link in your post doesn't work. But if I copy the text of the link and paste it into browser then it does work.

http://support.sas.com/resources/papers/Multilingual_Computing_with_SAS_94.pdf

Let me see if I can recreate the link to see why it didn't work in your post.

http://support.sas.com/resources/papers/Multilingual_Computing_with_SAS_94.pdf

So the link here works. I think in your post the URL had an extra space in it.

Re: Solution for ERROR: Some character data was lost during transcoding in the dataset

Ksharp — Fri, 31 Mar 2017 12:55:02 GMT


Yeah. That is ENCODING problem.
Change your sas session encoding to be the same encoding

Re: Solution for ERROR: Some character data was lost during transcoding in the dataset

ehbales — Fri, 31 Mar 2017 13:27:16 GMT

Thanks for pointing out the bad link! Sorry for the inconvenience.