topic Re: Identify duplicates in a file in SAS Enterprise Guide

Identify duplicates in a file

avatar — Fri, 22 Mar 2013 14:43:14 GMT

is there a way to identify duplicate ids in a file with 40,000 records?

Any suggestion would be greatly appreciated.

Thanks

Reeza — Fri, 22 Mar 2013 14:44:53 GMT

Task>Data>Sort> Under options, look for the first and the duplicate options.

Or look at proc sort.

avatar — Fri, 22 Mar 2013 15:45:05 GMT

Thanks a lot. I have few methods to try now,

avatar — Fri, 22 Mar 2013 17:18:57 GMT

data step using if not last.id then output dups;

else output unique; worked for me

Thanks

Reeza — Fri, 22 Mar 2013 17:33:28 GMT

it should be if not( last.id and first.id) to get both of the observations that are duplicate unless you want only the last one.

avatar — Fri, 22 Mar 2013 17:43:39 GMT

Interesting. I wanted both records. Now I got it . Thanks for the correction. Never knew about this code (if not last.id and first.id)

Haikuo — Fri, 22 Mar 2013 17:49:59 GMT

40,000 does not sound too much for Proc SQL. The following code may also help you reach your goal;

proc sql;

select * from yourdata group by ID having count(*)>1; quit;

Haikuo

Haris — Fri, 22 Mar 2013 17:51:53 GMT

This would print all the records for IDs with multiple entries:

proc sql;

select *

from data

group by ID

having sum(*) GT 1;

quit;