while reading data from a permanent dataset values are truncating for a specific column

[Mastanvali]

Hi Team,

I'm reading a permanent dataset but values are truncating for a specific column. Even after applying length or format or informat statements in SAS viya.

data abc;

set milk.apps;

run; 

milk.apps having product column and it contains length 20.

product column contains value "mástan o' valishaikka"

in ABC product column contains 

"mástan o' valishaikk".

may I know how sas reading special values?

Only truncating last character value in each record.

[hhh123]

##- Since you are reading multi byte character, using k functions like
klength instead of length should resolve the issue -##

[Mastanvali]

klength product $20.; /* it is throwing error*/

data d;

set milk.app;

asd=klength(product); /* it will give the length of the prodcut variable */

run;

I was tried but the statement is throwing error and it is used out of order.

any other suggestions.

[Tom]

---

@Mastanvali wrote:

klength product $20.; /* it is throwing error*/

data d;

set milk.app;

asd=klength(product); /* it will give the length of the prodcut variable */

run;

I was tried but the statement is throwing error and it is used out of order.

any other suggestions.

---

KLENGTH is NOT a statement.  There is a KLENGTH() function.

You need to put the LENGTH statement as part of the data step.  Variables are not some type of free standing object that is independent of the dataset they appear in.

Make the variable longer than you think you need and see if that helps.  Make sure to remove any format attached to the variable so the values are not truncated on printing.

data d;
  length product $40 ;
  format product ;
  set milk.app;
  nbytes = length(product);
  nchars = klength(product);
run;

[Mastanvali]

Even the value is truncating, set it self truncating. Any other suggestions?

[Tom]

Please confirm how the libref you are reading from is defined.  You mentioned before that you did not know the path. Why? Is it not coming from a SAS dataset?  Are you linking to some external database?  If so then perhaps the connection path for that is messing up the values before they even get to SAS.

If you do have a SAS dataset as input you can try doing your own transcoding.

I think something like this will let you get the raw values from the file and look at them.

You can use $HEX format to see the actual codes for charactes in the string. You can try using KCVT() function to transform the values between encodings.

data want ;
  set milk.app (encoding=any);
  length utf8 hex $200 ;
  hex = putn(product,cats('$hex,2*length(product),'.'));
  utf8 = kcvt(trim(product),'wlatin1','utf-8');
run;

[Mastanvali]

we are reading data from aws or some other external db. we are directly
using milk.app.

[Tom]

---

@Mastanvali wrote:
we are reading data from aws or some other external db. we are directly
using milk.app.

---

So you are NOT reading from a permanent SAS dataset?

How is the MILK libref defined? 

To seem some information run this statement.

libname milk list;

[Mastanvali]

Hey sorry for the inconvenience, those are secret libraries. So we don't
have path with us. those libraries configured in auto.exe files. we are
using directly.

[Astounding]

The values you are hoping to print actually contain 21 printable characters, not 20.  Have you tried just removing the format, along the lines of:

data abc;
   set milk.apps;
   format product;
run; 

[Mastanvali]

even after trying also values are getting truncated.

any other alternate way please?

[Tom]

Sounds like you need to use the CVP engine to expand the number of bytes of storage used to store the character strings so that it has enough space to store the multi-byte characters you are using.

https://documentation.sas.com/doc/en/vdmmlcdc/8.1/lestmtsref/p1ml1gmtqm3h72n17yjxbyagw0mm.htm

[Mastanvali]

I do not have the permanent path details with me to use the cvp.

please let me know the alternate way of the cvp engine instead of libanme.