Turn on suggestions

Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.

Showing results for

- Home
- /
- Programming
- /
- Enterprise Guide
- /
- Fuzzy grouping

Options

- RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page

- Mark as New
- Bookmark
- Subscribe
- Mute
- RSS Feed
- Permalink
- Report Inappropriate Content

Posted 10-21-2010 12:02 AM
(1095 views)

hello,

Can SAS performing fuzzy grouping?

i.e. I would like to find, within a data (say 8000 record and 200 columns), whether there are some pairs of data that are likely to be duplicates / similiars.

e.g.

id f1 f2 f3 f4 f5

1 1 2 3 4 5

2 2 3 4 5 6

3 3 4 5 6 7

4 3 4 7 8 9

5 1 3 3 3 3

6 1 2 3 4 5

aim: (a) find out the pairs of data which is exactly the same

(b) find out the pairs of data which is different in 3 columns or less

Result

(a) pair (1 - 6 ) with fields F1 to F5

(b) pair (3 - 4 ) with fields F3, F4, F5

pair (1 - 5 ) with fields F2, F4, F5

......

Thanks for your help

Can SAS performing fuzzy grouping?

i.e. I would like to find, within a data (say 8000 record and 200 columns), whether there are some pairs of data that are likely to be duplicates / similiars.

e.g.

id f1 f2 f3 f4 f5

1 1 2 3 4 5

2 2 3 4 5 6

3 3 4 5 6 7

4 3 4 7 8 9

5 1 3 3 3 3

6 1 2 3 4 5

aim: (a) find out the pairs of data which is exactly the same

(b) find out the pairs of data which is different in 3 columns or less

Result

(a) pair (1 - 6 ) with fields F1 to F5

(b) pair (3 - 4 ) with fields F3, F4, F5

pair (1 - 5 ) with fields F2, F4, F5

......

Thanks for your help

4 REPLIES 4

- Mark as New
- Bookmark
- Subscribe
- Mute
- RSS Feed
- Permalink
- Report Inappropriate Content

What you are talking about is, broadly, under the ETL (Extract, Transform, and Load) family of activities. DI may have some tools for that. You can accomplish your first goal in base SAS using SORT NODUPLICATES, or a SORT and DATA step if you want separate output files.

The second is tougher. I see lots of data sets with 3 variables alike but different subjects. A statistical approach might be to compute the Mahalanobis distances between pairs and look at ones the distance is below some threshold; it is a computer intensive approach. See

http://support.sas.com/kb/30/662.html

Doc Muhlbaier

Duke

The second is tougher. I see lots of data sets with 3 variables alike but different subjects. A statistical approach might be to compute the Mahalanobis distances between pairs and look at ones the distance is below some threshold; it is a computer intensive approach. See

http://support.sas.com/kb/30/662.html

Doc Muhlbaier

Duke

- Mark as New
- Bookmark
- Subscribe
- Mute
- RSS Feed
- Permalink
- Report Inappropriate Content

I think this is doable in a DATA step by reading the input data twice using two SET statements, the second in a loop using the POINT = option like so:

data example;

set inputdata;

do record = 1 to obsnum;

set inputdata point = record nobs = obsnum;

......checking statements......

end;

run;

By using this strategy you can compare every row with every other row in your data and test for exact or partial matches.

The trick for the checking logic would be to create two arrays so you can easily compare all 200 columns in a loop:

array vars (*) F1 - F200; * From first SET statement;

array vars2 (*) G1 - G200; * From second SET statement;

do i = 1 to dim(vars);

if vars (i) = vars2(i) then exact_match_count + 1;

end;

At the end of this logic you will know how many exact matches compared to total values for the pair of rows you are checking so you can write out a row to the result dataset the IDs for the row-pairs you are processing and the matching stats. Message was edited by: SASKiwi

data example;

set inputdata;

do record = 1 to obsnum;

set inputdata point = record nobs = obsnum;

......checking statements......

end;

run;

By using this strategy you can compare every row with every other row in your data and test for exact or partial matches.

The trick for the checking logic would be to create two arrays so you can easily compare all 200 columns in a loop:

array vars (*) F1 - F200; * From first SET statement;

array vars2 (*) G1 - G200; * From second SET statement;

do i = 1 to dim(vars);

if vars (i) = vars2(i) then exact_match_count + 1;

end;

At the end of this logic you will know how many exact matches compared to total values for the pair of rows you are checking so you can write out a row to the result dataset the IDs for the row-pairs you are processing and the matching stats. Message was edited by: SASKiwi

- Mark as New
- Bookmark
- Subscribe
- Mute
- RSS Feed
- Permalink
- Report Inappropriate Content

Thanks for your advice.

i have tried on the program, like:

data example;

set inputdata;

array vars (*) F1 - F200; * From first SET statement;

do record = 1 to obsnum;

set inputdata point = record nobs = obsnum;

array vars2 (*) G1 - G200; * From second SET statement;

do i = 1 to dim(vars);

if vars (i) = vars2(i) then exact_match_count + 1;

end;

end;

run;

but i found that there is a problem in the statement "if vars (i) = vars2(i) then exact_match_count + 1;", seems that the array vars(i) is not work.

Pls advice, Thanks

i have tried on the program, like:

data example;

set inputdata;

array vars (*) F1 - F200; * From first SET statement;

do record = 1 to obsnum;

set inputdata point = record nobs = obsnum;

array vars2 (*) G1 - G200; * From second SET statement;

do i = 1 to dim(vars);

if vars (i) = vars2(i) then exact_match_count + 1;

end;

end;

run;

but i found that there is a problem in the statement "if vars (i) = vars2(i) then exact_match_count + 1;", seems that the array vars(i) is not work.

Pls advice, Thanks

- Mark as New
- Bookmark
- Subscribe
- Mute
- RSS Feed
- Permalink
- Report Inappropriate Content

Sorry, I missed out the renaming of the variables on the second SET statement:

set inputdata (rename = (F1 - F200 = G1 - G200)) point = record nobs = obsnum;

This is needed so you can compare values between the different rows without overwriting and to populate the VARS2 array correctly so the exact matches can be counted.

set inputdata (rename = (F1 - F200 = G1 - G200)) point = record nobs = obsnum;

This is needed so you can compare values between the different rows without overwriting and to populate the VARS2 array correctly so the exact matches can be counted.

Build your skills. Make connections. Enjoy creative freedom. Maybe change the world. **Registration is now open through August 30th**. Visit the SAS Hackathon homepage.

SAS Enterprise Guide vs. SAS Studio

What’s the difference between SAS Enterprise Guide and SAS Studio? How are they similar? Just ask SAS’ Danny Modlin.

Find more tutorials on the SAS Users YouTube channel.