topic Re: while reading data from a permanent dataset values are truncating for a specific column in SAS Viya

while reading data from a permanent dataset values are truncating for a specific column

Mastanvali — Mon, 24 May 2021 14:52:27 GMT

Hi Team,

I'm reading a permanent dataset but values are truncating for a specific column. Even after applying length or format or informat statements in SAS viya.

data abc;

set milk.apps;

run;

milk.apps having product column and it contains length 20.

product column contains value "mástan o' valishaikka"

in ABC product column contains

"mástan o' valishaikk".

may I know how sas reading special values?

Only truncating last character value in each record.

Re: while reading data from a permanent dataset values are truncating for a specific column

hhh123 — Mon, 24 May 2021 15:22:06 GMT

##- Since you are reading multi byte character, using k functions like
klength instead of length should resolve the issue -##

Re: while reading data from a permanent dataset values are truncating for a specific column

Astounding — Mon, 24 May 2021 15:45:58 GMT

The values you are hoping to print actually contain 21 printable characters, not 20. Have you tried just removing the format, along the lines of:

data abc;
   set milk.apps;
   format product;
run;

Re: while reading data from a permanent dataset values are truncating for a specific column

Tom — Mon, 24 May 2021 15:59:22 GMT

Sounds like you need to use the CVP engine to expand the number of bytes of storage used to store the character strings so that it has enough space to store the multi-byte characters you are using.

https://documentation.sas.com/doc/en/vdmmlcdc/8.1/lestmtsref/p1ml1gmtqm3h72n17yjxbyagw0mm.htm

Re: while reading data from a permanent dataset values are truncating for a specific column

Mastanvali — Tue, 25 May 2021 08:27:41 GMT

I do not have the permanent path details with me to use the cvp.

please let me know the alternate way of the cvp engine instead of libanme.

Re: while reading data from a permanent dataset values are truncating for a specific column

Mastanvali — Tue, 25 May 2021 08:45:23 GMT

klength product $20.; /* it is throwing error*/

data d;

set milk.app;

asd=klength(product); /* it will give the length of the prodcut variable */

run;

I was tried but the statement is throwing error and it is used out of order.

any other suggestions.

Re: while reading data from a permanent dataset values are truncating for a specific column

Mastanvali — Tue, 25 May 2021 08:46:45 GMT

even after trying also values are getting truncated.

any other alternate way please?

Re: while reading data from a permanent dataset values are truncating for a specific column

Tom — Tue, 25 May 2021 12:00:07 GMT

@Mastanvali wrote:

klength product $20.; /* it is throwing error*/

data d;

set milk.app;

asd=klength(product); /* it will give the length of the prodcut variable */

run;

I was tried but the statement is throwing error and it is used out of order.

any other suggestions.

KLENGTH is NOT a statement. There is a KLENGTH() function.

You need to put the LENGTH statement as part of the data step. Variables are not some type of free standing object that is independent of the dataset they appear in.

Make the variable longer than you think you need and see if that helps. Make sure to remove any format attached to the variable so the values are not truncated on printing.

data d;
  length product $40 ;
  format product ;
  set milk.app;
  nbytes = length(product);
  nchars = klength(product);
run;

Re: while reading data from a permanent dataset values are truncating for a specific column

Mastanvali — Tue, 25 May 2021 12:30:06 GMT

Even the value is truncating, set it self truncating. Any other suggestions?

Re: while reading data from a permanent dataset values are truncating for a specific column

Tom — Tue, 25 May 2021 12:43:02 GMT

Please confirm how the libref you are reading from is defined. You mentioned before that you did not know the path. Why? Is it not coming from a SAS dataset? Are you linking to some external database? If so then perhaps the connection path for that is messing up the values before they even get to SAS.

If you do have a SAS dataset as input you can try doing your own transcoding.

I think something like this will let you get the raw values from the file and look at them.

You can use $HEX format to see the actual codes for charactes in the string. You can try using KCVT() function to transform the values between encodings.

data want ;
  set milk.app (encoding=any);
  length utf8 hex $200 ;
  hex = putn(product,cats('$hex,2*length(product),'.'));
  utf8 = kcvt(trim(product),'wlatin1','utf-8');
run;

Re: while reading data from a permanent dataset values are truncating for a specific column

Mastanvali — Tue, 25 May 2021 12:50:06 GMT

we are reading data from aws or some other external db. we are directly
using milk.app.

Re: while reading data from a permanent dataset values are truncating for a specific column

Tom — Tue, 25 May 2021 14:30:00 GMT

@Mastanvali wrote:
we are reading data from aws or some other external db. we are directly
using milk.app.

So you are NOT reading from a permanent SAS dataset?

How is the MILK libref defined?

To seem some information run this statement.

libname milk list;

Re: while reading data from a permanent dataset values are truncating for a specific column

Mastanvali — Tue, 25 May 2021 16:40:06 GMT

Hey sorry for the inconvenience, those are secret libraries. So we don't
have path with us. those libraries configured in auto.exe files. we are
using directly.