topic Re: MD5 with cats function in SAS Programming

MD5 with cats function

langalife — Wed, 02 Sep 2020 10:11:19 GMT

Good day

i have a md5 macro that concatenates variables which I call later on,however the lenght is kept at 200, i would like to know how extend the lenght here is the sample code below.

%macro md5_concatenate_mydata();
md5(
cats(
ACCOUNT_NUMBER,'|',

put(balance, best32.) ,'|',

status ,'|',

)

);

%mend md5_concatenate_mydata;

say up to 50 variables, now i call it in a data step, it only keeps 200 characters not more when i call it

how can i format it to go beyond 200 in lenght? as it is based on concatination of the md5 macro above.

%macro hashtables( i_hashFunction=, i_table=, i_primaryKey= );

data work.data_want (keep = account_number md5 );
set &i_table;

md5 = %&i_hashFunction.();

run;

%mend hashtables;

%hashtables(
i_hashFunction = md5_concatenate_mydata,
i_table = data_want.,
i_primarykey = ACCOUNT_NUMBER );

Re: MD5 with cats function

andreas_lds — Wed, 02 Sep 2020 10:24:37 GMT

Have a look at the documentation:

In a DATA step, if the CATS function returns a value to a variable that has not previously been assigned a length, that variable is given a length of 200 bytes.

Re: MD5 with cats function

langalife — Wed, 02 Sep 2020 10:31:00 GMT

i had a look at documentation, it didnt help.

if that was a variable it was going to be easy for me to change.

Re: MD5 with cats function

andreas_lds — Wed, 02 Sep 2020 12:03:38 GMT

@langalife wrote:

Hi

i had a look at documentation, it didnt help.

if that was a variable it was going to be easy for me to change.

ok. back to square one: i would try:

data work.data_want (keep = account_number md5 );
  set  &i_table;
  length allVars $ 1000;
  allVars = cats(list_vars_skipping_separating_chars_and_formats);
  md5 = md5(allVars);
run;

Re: MD5 with cats function

Quentin — Wed, 02 Sep 2020 13:07:22 GMT

I'm confused. In the DATA step (outside of a WHERE statement) the buffer for CATS should be 32,767 per the docs:

https://documentation.sas.com/?docsetId=lefunctionsref&docsetTarget=n0p7wxtk0hvn83n1pveisbcp2ae9.htm&docsetVersion=9.4&locale=en

Below test seems to confirm this:

data have ;
  retain x1-x33000 'X' ;
run ;

data want0;
  set  have;
  md5 = md5(cats(of x1-x32767));
run;

data want1;
  set  have;
  md5 = md5(cats(of x1-x32768));
run;

data want2;
  set  have;
  md5 = md5(cats(of x1-x32769));
run;

proc compare base=want0 compare=want1 ;
run ;

proc compare base=want1 compare=want2 ;
run ;

Do you think your real data exceed the 32,767 limit?

In the log, are you getting the warning:

WARNING: In a call to the CATS function, the buffer allocated for the result was not long enough
         to contain the concatenation of all the arguments. The correct result would contain
         32768 characters, but the actual result might either be truncated to 32767 character(s)
         or be completely blank, depending on the calling environment. The following note
         indicates the left-most argument that caused truncation.

Re: MD5 with cats function

jimbarbour — Wed, 02 Sep 2020 14:52:24 GMT

@langalife,

Perhaps I'm missing something, but have you tried adding a length in your Data step to the md5 variable?

data work.data_want (keep = account_number md5 );
     LENGTH md5 $32767;
     set  &i_table;

     md5 = %&i_hashFunction.();
run;

If md5 is already defined in the data coming in via the Set statement, you may have to either modify the dataset or rename the variable (using the RENAME option on the Set statement) to some new name, say "temp_md5", and then move the value of temp_md5 to md5, i.e.:

md5 = temp_md5;

Jim

Re: MD5 with cats function

Reeza — Wed, 02 Sep 2020 16:57:26 GMT

Rules vary for CAT depending on where it's being used.

The CATX function returns a value to a variable or returns a value in a temporary buffer. The value that is returned from the CATX function has one of these lengths:

up to 200 characters in WHERE clauses and in PROC SQL
up to 32767 characters in the DATA step, except in WHERE clauses
up to 65534 characters when CATX is called from the macro processor

FYI - it looks like you're actually building a function, could PROC FCMP be used here instead? it's a slightly cleaner approach.

Re: MD5 with cats function

Quentin — Thu, 03 Sep 2020 13:22:50 GMT

Thanks @Reeza. In the OP's example, %hashtables, they're using it in DATA step outside of a where clause, which is why I said they should be getting the full 32,767 characters. But if IRL they're using on a WHERE clause, that would explain why they're getting so much truncation. And I agree, likely FCMP could be a better route.

Re: MD5 with cats function

langalife — Mon, 07 Sep 2020 13:52:01 GMT

Upto so far i havent found a posted solution.

I use the cats function inside md5, example is as shown below.

%macro hashtables( i_hashFunction=, i_table=, i_primaryKey= );

data work.data_want (keep = account_number md5 );
set &i_table;

md5 =

md5(
cats(
ACCOUNT_NUMBER,'|',

put(balance, best32.) ,'|',

status ,'|',

)

);

run;

%mend hashtables;

%hashtables(
i_table = data_want.,
i_primarykey = ACCOUNT_NUMBER );

Re: MD5 with cats function

Quentin — Mon, 07 Sep 2020 18:56:38 GMT

You didn't answer my question. In the log, are you getting the below warning (or a similar warning)?

WARNING: In a call to the CATS function, the buffer allocated for the result was not long enough
         to contain the concatenation of all the arguments. The correct result would contain
         32768 characters, but the actual result might either be truncated to 32767 character(s)
         or be completely blank, depending on the calling environment. The following note
         indicates the left-most argument that caused truncation.

Can you post your log?

Re: MD5 with cats function

ChrisNZ — Wed, 09 Sep 2020 21:15:45 GMT

Adding the put function should lengthen your variable (untested):

%macro md5_concatenate_mydata();
  put(md5(catx( '|'
              , ACCOUNT_NUMBER
              , put(BALANCE, best32.)  
              , STATUS
              ,
              ,
     )), $32000.);
%mend md5_concatenate_mydata;

Re: MD5 with cats function

ChrisNZ — Wed, 09 Sep 2020 21:16:39 GMT

Note that MD5 is now considered insecure, and that SHA256 is to be preferred.