topic Re: Duplicate record by similarity score/closer value in SAS Data Management

Duplicate record by similarity score/closer value

mlogan — Thu, 23 Aug 2018 19:22:17 GMT

Hi All,

I am looking for a code that will allow me to find duplicate based on exact as well as similar value. I know for exact value we can use NODUP, NODUPKEYS and NOUNIQUEKEYS. But can anyone tell me what function I can use to identify duplicate records based on similarity value.

Example: Following two records should be considered as duplicate though Last name and Address are not same (but they are similar).

First Name Last Name Address

John Murruy 1 New York St.

John Murray 1 New York Street

Thanks,

Re: Duplicate record by similarity score/closer value

PeterClemmensen — Thu, 23 Aug 2018 21:01:50 GMT

This question is a bit fuzzy in its nature because how similar should two strings be for them to be considered equal? Ie. when are two strings in two different observations similar enough to be considered duplicates and therefore omitted?

Two functions to get you going are the COMPLEV and COMPGED Functions. Both functions take two strings as input and return a number, which represents the 'distance' between two strings.

Re: Duplicate record by similarity score/closer value

SimonDawson — Fri, 24 Aug 2018 10:49:22 GMT

If you have a license for the SAS Data Quality procedures I'd look into match code generation to solve this type of problem.

http://support.sas.com/documentation/cdl/en/dqclref/70016/HTML/default/viewer.htm#n1597gcbsehaokn1j500v6ius99p.htm