topic Re: Why the merged data has more obs than before in SAS Programming

Why the merged data has more obs than before

LarissaW — Wed, 26 Apr 2023 00:28:23 GMT

I was trying to merge two datasets and I used proc compare to check before I merged them, below is the result from proc compare

The code I used to merge datasets is

proc sort data=LC.FAM;
by pid;
run;

proc sort data=hepa;
by PID;
run;

data combined1;
    merge LC.FAM  hepa ;
    by PID;
run;

I got a new dataset with 50273 rows, which contains more rows than hepa (50032 obs). Did anyone know why this happens?

Re: Why the merged data has more obs than before

Quentin — Wed, 26 Apr 2023 00:43:16 GMT

This can happen if there are duplicates in the data, or if there are mis-matches (e.g. values for pid in lc.fam that are not in hepa).

Running PROC COMPARE is an interesting idea. You usually use PROC COMPARE to compare variables, but it will also tell you if you have duplicate values, or if you have mis-matches. What do you get if you run:

proc compare base=hepa compare=lc.fam ;
  id pid ; *use an ID statement here, not a BY statement;
run ;

Re: Why the merged data has more obs than before

Tom — Wed, 26 Apr 2023 03:14:17 GMT

There is no way to know in advance how many observations the merge will generate. Unless you know the values of PID in both datasets.

If HEPA has 100 observations that are each a distinct value of PID

And FAM has 10 observations that are each a distinct value of PID

Then merging them can result in between 100 observations (all of the values of PID in FAM already existed in HEPA) to 110 observations (none of the values of PID in FAM existed in HEPA).

And if either dataset has multiple observations for the same PID then even stranger things can happen.

Re: Why the merged data has more obs than before

Kurt_Bremser — Wed, 26 Apr 2023 05:18:14 GMT

This has nothing to do with graphics, so I moved this to the general Programming community.

Re: Why the merged data has more obs than before

A_Kh — Wed, 26 Apr 2023 13:35:19 GMT

In addition to abovementioned comments, if you need to merge based on PID.HEPA then use conditional merge, or SQL joins, then you get the 50032 obs.

eg:

data combined1;
    merge LC.FAM (in=a)  hepa (in=b);
    by PID;
    if b;
run;