topic Understanding the Fuzzy Matching Techniques Behind Matchcode Generation in SAS Data Quality in SAS Data Management

Understanding the Fuzzy Matching Techniques Behind Matchcode Generation in SAS Data Quality

saunvida — Fri, 20 Dec 2024 06:30:22 GMT

Hello,

I have been researching various operations that can be performed on textual data in SAS Data Quality and am particularly interested in the Matching operation. While going through the DataFlux documentation, I have gained a fair understanding of concepts like match definitions, schemes, and chop tables.

However, I am curious about the fuzzy matching techniques used to generate matchcodes. Specifically, I would like to understand:

What fuzzy matching techniques or algorithms are used in SAS Data Quality for matchcode generation?
Are these techniques based on phonetic algorithms (e.g., Soundex) or string similarity measures (e.g., Levenshtein distance, Jaro-Winkler)?

I understand the what of match definitions and matchcodes, but now I am keen to dive deeper into the how. Any guidance or references would be greatly appreciated!

Thank you in advance!

Re: Understanding the Fuzzy Matching Techniques Behind Matchcode Generation in SAS Data Quality

Rama_V — Tue, 07 Jan 2025 00:51:43 GMT

I hope you are still interested in this topic.

The QKB is used in generating Fuzzy Codes. It is a collection of file definitions, schemas, chop tables, phonetic libraries, regex Libraries, vocabularies, and grammars. These files can be edited in the Data Flux Management Studio Application. If out-of-the-box rules don't meet your organisation's needs then you can add or edit files. For me, I struggled with vocabulary and grammar files.

Match code generation follows a series of steps, using the rules from those files to tidy the string at each step to remove noise, standardize, normalize, phonetic reduction, and create a Matchcode Layout. It is a lot more than just the Soundex function.

QKB Definition Steps

The sensitivity defines the number of characters used to create a fuzzy code.

Sensitivity

Finally, MatchCode is an unencoded string, that is converted to is encoded Fuzzy String. The encoding logic is hidden but the node generates it based on characters.
Encoding

For each value in the column data, these steps are followed to generate match codes. For a data value, Matchcodes do not change unless the QKB version or rules are edited. Hope this gives some understanding of the generation of Match Codes.

If you are interested in learning and editing those files then there are QKB courses on SAS Learning,
Wish you luck.

Rama

Re: Understanding the Fuzzy Matching Techniques Behind Matchcode Generation in SAS Data Quality

SASKiwi — Tue, 07 Jan 2025 02:10:36 GMT

It's also worth understanding that SAS Data Quality's QKBs are country-specific. If you are trying to match addresses say in India then you need the Indian QKB which is customised to deal with the unique Indian address formatting.