Merging two data sets and ensuring additional records are not created.

[Paul_NYS]

Hi Everyone

I am merging two data sets with the below statement and when I do a few extra records are being created in the new data set. Is there any way I can either prevent this or identify the additional records in the new data set?

In the s0ssrp data set, there are 514,983. In the new s0ssrp1, there are 515,013.

Paul

data s0ssrp1 (rename=closed_reason=FtcOutcome);
merge s0ssrp (in=a) ssrpucms2 (keep=County Docket ReferralDate FtcStatus closed_reason ContractDate);
if a;
by County Docket;

if ReferralDate ge start and ReferralDate le stop then do;
FtcReferred="Y";
end;

else FtcReferred="N";
run;

[Astounding]

This would happen if your second data set, ssrpucms2, occasionally contained more than one observation for a combination of COUNTY + DOCKET. 

Do you need help in locating where the duplicates are?

[Paul_NYS]

I thought there were a lot of duplicates actually. But if this is the only scenario that would produce more than one record from the original data set, then I will take a look at the file tomorrow morning and check it further. If that is the case then that is an easier scenario than I thought.

I will get back. Thanks Astounding.

Paul

[FreelanceReinh]

Hi Paul,

You can find the duplicate combinations of County and Docket in dataset SSRPUCMS2 by submitting the following data step:

data dups;
set ssrpucms2;
by County Docket;
if ~(first.docket & last.docket);
run;

If this resulted in more than 60 (=2*(515,013-514,983)) observations, it would seem that also dataset S0SSRP contains such duplicates. In this case the merge step would be questionable and should have created a message

NOTE: MERGE statement has more than one data set with repeats of BY values.

in the log. 

So, you could replace ssrpucms2 by s0ssrp in the above data step and should be alarmed if the resulting dataset DUPS is not empty.

[PGStats]

If County-Docket is a unique key in s0ssrp, you can choose to use the first or the last matching record from ssrpucms2 in the following way:

data s0ssrp1 (rename=closed_reason=FtcOutcome);
merge s0ssrp (in=a) ssrpucms2 (in=b keep=County Docket ReferralDate FtcStatus closed_reason ContractDate);
by County Docket;

/* Use the first record from ssrpucms2 in a County-Docket group */
if a and (not b or first.Docket);

/* Or... */

/* Use the last record from ssrpucms2 in a County-Docket group */
if a and (not b or last.Docket);

if ReferralDate ge start and ReferralDate le stop then FtcReferred="Y";
else FtcReferred="N";

run;