topic Fuzzy logic to locate duplicates? in SAS Programming

Fuzzy logic to locate duplicates?

Doug____ — Mon, 12 Feb 2018 17:56:17 GMT

How can fuzzy logic be used to locate duplicate records in a single dataset? I have records where spellings can vary (or be slightly incorrect) based on a combination of free text and standard text in some variables, so the standard NODUPKEY or sort with FIRST or LAST in data steps do not work correctly.

Re: Fuzzy logic to locate duplicates?

Reeza — Mon, 12 Feb 2018 18:26:18 GMT

This is a non-trivial problem.

One factor is the number of comparisons required, because you're now required to compare each record to all other records you're trying to match to, so it becomes a N*M comparison.

You can look at the following functions for starter:

COMPGED

SOUNDEX/SOUNDS LIKE

@Doug____ wrote:

How can fuzzy logic be used to locate duplicate records in a single dataset? I have records where spellings can vary (or be slightly incorrect) based on a combination of free text and standard text in some variables, so the standard NODUPKEY or sort with FIRST or LAST in data steps do not work correctly.

Re: Fuzzy logic to locate duplicates?

SuryaKiran — Mon, 12 Feb 2018 22:29:27 GMT

I prefer using SPEDIS() function which gives a score on the likelihood of two variables.

SPEDIS(First_name,First_name_match)<5;

COMPGED is also similar function.

Re: Fuzzy logic to locate duplicates?

ballardw — Tue, 13 Feb 2018 00:25:56 GMT

How many records are you looking at?

Are there other variables in the data that help identify individuals (date of birth, address, other fields?)

Re: Fuzzy logic to locate duplicates?

ChrisNZ — Tue, 13 Feb 2018 02:22:26 GMT

I found COMPGED gives more usable results than SPEDIS.

Also, a good way to limit the size of the cartesian products, if applicable, it to match on similar lengths and on identical starting letter(s).