Duplicate record by similarity score/closer value

[mlogan]

Hi All,

I am looking for a code that will allow me to find duplicate based on exact as well as similar value. I know for exact value we can use NODUP, NODUPKEYS and NOUNIQUEKEYS. But can anyone tell me what function I can use to identify duplicate records based on similarity value.

Example: Following two records should be considered as duplicate though Last name and Address are not same (but they are similar).  

First Name      Last Name  Address

John                Murruy         1 New York St.

John                Murray         1 New York Street 

Thanks,

[PeterClemmensen]

This question is a bit fuzzy in its nature because how similar should two strings be for them to be considered equal? Ie. when are two strings in two different observations similar enough to be considered duplicates and therefore omitted?

Two functions to get you going are the COMPLEV and COMPGED Functions. Both functions take two strings as input and return a number, which represents the 'distance' between two strings. 

[SimonDawson]

If you have a license for the SAS Data Quality procedures I'd look into match code generation to solve this type of problem.

http://support.sas.com/documentation/cdl/en/dqclref/70016/HTML/default/viewer.htm#n1597gcbsehaokn1j5...