MD5 with cats function

[langalife]

Good day 

i have a md5 macro that concatenates variables which I call later on,however the lenght is kept at 200, i would like to know how extend the lenght here is the sample code below.

%macro md5_concatenate_mydata();
md5(
cats(
ACCOUNT_NUMBER,'|',

put(balance, best32.)  ,'|',

status ,'|',

.

.

.

.

.

)

);

%mend md5_concatenate_mydata;

say up to 50 variables, now i call it in a data step, it only  keeps 200 characters not more when i call it

how can i format it to go beyond 200 in lenght? as it is based on concatination of the md5 macro above.

%macro hashtables( i_hashFunction=, i_table=, i_primaryKey= );

data work.data_want (keep = account_number md5 );
set  &i_table;

md5 = %&i_hashFunction.();

run;

%mend hashtables;

%hashtables(
i_hashFunction = md5_concatenate_mydata,
i_table = data_want.,
i_primarykey = ACCOUNT_NUMBER );

[andreas_lds]

Have a look at the documentation:

In a DATA step, if the CATS function returns a value to a variable that has not previously been assigned a length, that variable is given a length of 200 bytes.

[langalife]

Hi 

i had a look at documentation, it didnt help.

if that was a variable it was going to be easy for me to change.

[andreas_lds]

---

@langalife wrote:

Hi 

i had a look at documentation, it didnt help.

if that was a variable it was going to be easy for me to change.

---

ok. back to square one: i would try:

data work.data_want (keep = account_number md5 );
  set  &i_table;
  length allVars $ 1000;
  allVars = cats(list_vars_skipping_separating_chars_and_formats);
  md5 = md5(allVars);
run;

[Quentin]

I'm confused.  In the DATA step (outside of a WHERE statement) the buffer for CATS should be 32,767 per the docs:

https://documentation.sas.com/?docsetId=lefunctionsref&docsetTarget=n0p7wxtk0hvn83n1pveisbcp2ae9.htm...

Below test seems to confirm this:

data have ;
  retain x1-x33000 'X' ;
run ;

data want0;
  set  have;
  md5 = md5(cats(of x1-x32767));
run;

data want1;
  set  have;
  md5 = md5(cats(of x1-x32768));
run;

data want2;
  set  have;
  md5 = md5(cats(of x1-x32769));
run;

proc compare base=want0 compare=want1 ;
run ;

proc compare base=want1 compare=want2 ;
run ;

Do you think your real data exceed the 32,767 limit?

In the log, are you getting the warning:

WARNING: In a call to the CATS function, the buffer allocated for the result was not long enough
         to contain the concatenation of all the arguments. The correct result would contain
         32768 characters, but the actual result might either be truncated to 32767 character(s)
         or be completely blank, depending on the calling environment. The following note
         indicates the left-most argument that caused truncation.

[Reeza]

Rules vary for CAT depending on where it's being used. 

The CATX function returns a value to a variable or returns a value in a temporary buffer. The value that is returned from the CATX function has one of these lengths:

• up to 200 characters in WHERE clauses and in PROC SQL
• up to 32767 characters in the DATA step, except in WHERE clauses
• up to 65534 characters when CATX is called from the macro processor

FYI - it looks like you're actually building a function, could PROC FCMP be used here instead? it's a slightly cleaner approach. 

[Quentin]

Thanks @Reeza. In the OP's example, %hashtables, they're using it in DATA step outside of a where clause, which is why I said they should be getting the full 32,767 characters. But if IRL they're using on a WHERE clause, that would explain why they're getting so much truncation. And I agree, likely FCMP could be a better route.

[jimbarbour]

@langalife,

Perhaps I'm missing something, but have you tried adding a length in your Data step to the md5 variable?

data work.data_want (keep = account_number md5 );
     LENGTH md5 $32767;
     set  &i_table;

     md5 = %&i_hashFunction.();
run;

If md5 is already defined in the data coming in via the Set statement, you may have to either modify the dataset or rename the variable (using the RENAME option on the Set statement) to some new name, say "temp_md5",  and then move the value of temp_md5 to md5, i.e.:

md5 = temp_md5;

Jim

[langalife]

Upto so far i havent found a posted solution.

I use the cats function inside md5, example is as shown below.

%macro hashtables( i_hashFunction=, i_table=, i_primaryKey= );

data work.data_want (keep = account_number md5 );
set  &i_table;

md5 = 

md5(
cats(
ACCOUNT_NUMBER,'|',

put(balance, best32.)  ,'|',

status ,'|',

.

.

.

.

.

)

);

run;

%mend hashtables;

%hashtables(
i_table = data_want.,
i_primarykey = ACCOUNT_NUMBER );

[Quentin]

You didn't answer my question. In the log, are you getting the below warning (or a similar warning)?

WARNING: In a call to the CATS function, the buffer allocated for the result was not long enough
         to contain the concatenation of all the arguments. The correct result would contain
         32768 characters, but the actual result might either be truncated to 32767 character(s)
         or be completely blank, depending on the calling environment. The following note
         indicates the left-most argument that caused truncation.

Can you post your log?

[ChrisNZ · Accepted solution]

Adding the put function should lengthen your variable (untested):

%macro md5_concatenate_mydata();
  put(md5(catx( '|'
              , ACCOUNT_NUMBER
              , put(BALANCE, best32.)  
              , STATUS
              ,
              ,
     )), $32000.);
%mend md5_concatenate_mydata;

[ChrisNZ]

Note that MD5 is now considered insecure, and that SHA256 is to be preferred.