topic Re: Identifying fuzzy matches within a single variable in SAS Programming

Identifying fuzzy matches within a single variable

Walternate — Thu, 18 May 2023 15:06:22 GMT

Hi all,

I have a dataset which is a directory listing produced using a pipe statement. It has folder and file names.

What I'm trying to get at specifically is that there are some cases where the file name is close, but not identical.

Think:

clinic_file_clin1.csv

clinic_file_clin1_c.csv

So clinic_file_ is the beginning of all the file names, clin# is the clinic name (should only be one per directory). The difference will be something at the end like _c, _copy, _fixed, or something like that.

I'm trying to identify the files that occur in the directory more than once.

What is the best approach for this? I assume fuzzy matching of some kind, but from poking around it seems like that is usually across two columns/while matching.

Re: Identifying fuzzy matches within a single variable

Reeza — Thu, 18 May 2023 15:16:44 GMT

You can self join a dataset, if you know it will always be the same with a suffix, you can use like.

Re: Identifying fuzzy matches within a single variable

ballardw — Thu, 18 May 2023 15:19:50 GMT

"I'm trying to identify the files that occur in the directory more than once. " Easy: none.

SIMILARLY named is valid question.

The functions COMPGED and similar would be a place to start.

Here is a small example generating three different scores. Lower score are "more similar" within one of the comparisons.

data have;
   input str $;
datalines;
abc
abc1
abc_1
abc*
pdq
pdc1
;

proc sql;
   create table example as
   select a.str as stringa,b.str as stringb
         ,compged(a.str,b.str) as compgedscore         
         ,complev(a.str,b.str) as complevscore
         ,spedis(a.str,b.str) as spedisscore
   from have as a, have as b
   where a.str ne b.str
   ;
run;

Look at some of your values and you could set a threshold for a specific scoring method to select likely "similar" names.

Read the documentation for the functions. I'm not going to repeat paragraphs of details.

Not the you can create custom scoring rules for COMPGED using the CALL COMPCOST routine but likely not worth the effort unless you have hundreds of similarity patterns to look for.

Re: Identifying fuzzy matches within a single variable

Reeza — Thu, 18 May 2023 16:20:35 GMT

If this is only at the end, then something like this:

data have;
input file_name :$50.;
cards;
clinic_file_clin1.csv
clinic_file_clin1_c.csv
clinic_file_clin2.csv
clinic_file_clin2_fixed.csv
;

proc sql;
create table want as
select t1.file_name, t2.file_name as possible_duplicate
from have as t1
cross join have as t2
where find(scan(t1.file_name, 1, "."), scan(t2.file_name, 1, "."), 'it') >0 and t1.file_name ne t2.file_name;
quit;