Help using Base SAS procedures

Merging two data sets and ensuring additional records are not created.

Reply
Regular Contributor
Posts: 216

Merging two data sets and ensuring additional records are not created.

Hi Everyone

I am merging two data sets with the below statement and when I do a few extra records are being created in the new data set. Is there any way I can either prevent this or identify the additional records in the new data set?

 

In the s0ssrp data set, there are 514,983. In the new s0ssrp1, there are 515,013.

 

Paul

 

data s0ssrp1 (rename=closed_reason=FtcOutcome);
merge s0ssrp (in=a) ssrpucms2 (keep=County Docket ReferralDate FtcStatus closed_reason ContractDate);
if a;
by County Docket;

if ReferralDate ge start and ReferralDate le stop then do;
FtcReferred="Y";
end;

else FtcReferred="N";
run;

Super User
Posts: 5,076

Re: Merging two data sets and ensuring additional records are not created.

This would happen if your second data set, ssrpucms2, occasionally contained more than one observation for a combination of COUNTY + DOCKET. 

 

Do you need help in locating where the duplicates are?

Regular Contributor
Posts: 216

Re: Merging two data sets and ensuring additional records are not created.

I thought there were a lot of duplicates actually. But if this is the only scenario that would produce more than one record from the original data set, then I will take a look at the file tomorrow morning and check it further. If that is the case then that is an easier scenario than I thought.

 

I will get back. Thanks Astounding.

 

Paul

Trusted Advisor
Posts: 1,115

Re: Merging two data sets and ensuring additional records are not created.

Hi Paul,

 

You can find the duplicate combinations of County and Docket in dataset SSRPUCMS2 by submitting the following data step:

data dups;
set ssrpucms2;
by County Docket;
if ~(first.docket & last.docket);
run;

If this resulted in more than 60 (=2*(515,013-514,983)) observations, it would seem that also dataset S0SSRP contains such duplicates. In this case the merge step would be questionable and should have created a message

NOTE: MERGE statement has more than one data set with repeats of BY values.

in the log. 

 

So, you could replace ssrpucms2 by s0ssrp in the above data step and should be alarmed if the resulting dataset DUPS is not empty.

Respected Advisor
Posts: 4,641

Re: Merging two data sets and ensuring additional records are not created.

If County-Docket is a unique key in s0ssrp, you can choose to use the first or the last matching record from ssrpucms2 in the following way:

 

data s0ssrp1 (rename=closed_reason=FtcOutcome);
merge s0ssrp (in=a) ssrpucms2 (in=b keep=County Docket ReferralDate FtcStatus closed_reason ContractDate);
by County Docket;

/* Use the first record from ssrpucms2 in a County-Docket group */
if a and (not b or first.Docket);

/* Or... */

/* Use the last record from ssrpucms2 in a County-Docket group */
if a and (not b or last.Docket);

if ReferralDate ge start and ReferralDate le stop then FtcReferred="Y";
else FtcReferred="N";

run;
PG
Ask a Question
Discussion stats
  • 4 replies
  • 330 views
  • 1 like
  • 4 in conversation