BookmarkSubscribeRSS Feed
🔒 This topic is solved and locked. Need further help from the community? Please sign in and ask a new question.
Doug____
Pyrite | Level 9

How can fuzzy logic be used to locate duplicate records in a single dataset? I have records where spellings can vary (or be slightly incorrect) based on a combination of free text and standard text in some variables, so the standard NODUPKEY or sort with FIRST or LAST in data steps do not work correctly.

1 ACCEPTED SOLUTION

Accepted Solutions
SuryaKiran
Meteorite | Level 14

I prefer using SPEDIS() function which gives a score on the likelihood of two variables. 

 

SPEDIS(First_name,First_name_match)<5;

 

COMPGED is also similar function. 

Thanks,
Suryakiran

View solution in original post

4 REPLIES 4
Reeza
Super User

This is a non-trivial problem.  

 

One factor is the number of comparisons required, because you're now required to compare each record to all other records you're trying to match to, so it becomes a N*M comparison. 

 

You can look at the following functions for starter:

 

COMPGED

SOUNDEX/SOUNDS LIKE

 


@Doug____ wrote:

How can fuzzy logic be used to locate duplicate records in a single dataset? I have records where spellings can vary (or be slightly incorrect) based on a combination of free text and standard text in some variables, so the standard NODUPKEY or sort with FIRST or LAST in data steps do not work correctly.


 

 

 

SuryaKiran
Meteorite | Level 14

I prefer using SPEDIS() function which gives a score on the likelihood of two variables. 

 

SPEDIS(First_name,First_name_match)<5;

 

COMPGED is also similar function. 

Thanks,
Suryakiran
ballardw
Super User

How many records are you looking at?

Are there other variables in the data that help identify individuals (date of birth, address, other fields?)

 

 

ChrisNZ
Tourmaline | Level 20

I found COMPGED gives more usable results than SPEDIS. 

Also, a good way to limit the size of the cartesian products, if applicable, it to match on similar lengths and on identical starting letter(s).

 

sas-innovate-2024.png

Join us for SAS Innovate April 16-19 at the Aria in Las Vegas. Bring the team and save big with our group pricing for a limited time only.

Pre-conference courses and tutorials are filling up fast and are always a sellout. Register today to reserve your seat.

 

Register now!

How to Concatenate Values

Learn how use the CAT functions in SAS to join values from multiple variables into a single value.

Find more tutorials on the SAS Users YouTube channel.

Click image to register for webinarClick image to register for webinar

Classroom Training Available!

Select SAS Training centers are offering in-person courses. View upcoming courses for:

View all other training opportunities.

Discussion stats
  • 4 replies
  • 1283 views
  • 4 likes
  • 5 in conversation