topic Re: Deleting duplicates based on multiple criteria in SAS Procedures

Deleting duplicates based on multiple criteria

lboyd — Tue, 10 Oct 2017 21:47:12 GMT

How can I delete duplicates of one variable based on a start date using NODUPKEY?

Thanks!

Re: Deleting duplicates based on multiple criteria

Reeza — Tue, 10 Oct 2017 22:38:27 GMT

Provide more details.

@lboyd wrote:

How can I delete duplicates of one variable based on a start date using NODUPKEY?

Thanks!

Re: Deleting duplicates based on multiple criteria

lboyd — Wed, 11 Oct 2017 13:42:51 GMT

I've been using this:
PROC SORT DATA=CMpre DUPOUT=results NODUPKEY ;
BY QID25;
RUN ;

where qid25 is an email address. I need to get rid of duplicate email addresses and I want to keep the one that had the earliest start date. Startdate variable looks something like this:
22JUN17:00:00:00

Re: Deleting duplicates based on multiple criteria

Reeza — Wed, 11 Oct 2017 14:52:37 GMT

Sort it twice. First time with email address and date, second with email address and the NODUPKEY option.

Re: Deleting duplicates based on multiple criteria

lboyd — Wed, 11 Oct 2017 19:46:30 GMT

This code gets rid of missing e-mails, is there a way to prevent that?

Re: Deleting duplicates based on multiple criteria

lboyd — Wed, 11 Oct 2017 20:12:53 GMT

Also some of the e-mails start with an uppercase letter while others start with lower case-is there any way to delete based on both? For instance, if someone said Sam123@gmail.com and also sam123@gmail.com-I'd want one of those deleted from the database.

Re: Deleting duplicates based on multiple criteria

Reeza — Mon, 16 Oct 2017 14:52:04 GMT

For instance, if someone said Sam123@gmail.com and also sam123@gmail.com-I'd want one of those deleted from the database.

To fix this you need to clean your data first.

This code gets rid of missing e-mails, is there a way to prevent that?

To deal with this you likely need to do it manually.

First sort and then use a data step with first/last but coding an exception for the missing emails.

proc sort data=have;
by group_var;
run;

data want;
set have;
by group_var;
if first.group_var or missing(group_var);
run;

Re: Deleting duplicates based on multiple criteria

lboyd — Mon, 16 Oct 2017 14:52:57 GMT

Thank you!