DATA Step, Macro, Functions and more

Possible to merge by dates within a range?

Reply
Frequent Contributor
Posts: 78

Possible to merge by dates within a range?

I have two datasets that need merged together and even though they have an ID number, I need some other variable that will match them since they can have multiple records. My only option is to use a date variable. The catch is that the date variable will not be the same exact date, but should fall within a range of +/- 30 days.

I have the following merge variables in each dataset:

id examdate;

id examdate examdate_p30 examdate_m30;

examdate matches in 70% of the population, but I need to match the other 30% allowing examdate in dataset 1 to fall between examdate_p30 and examdate_m30 from dataset 2.

Any thoughts?

Thanks in advance.
Trusted Advisor
Posts: 2,113

Re: Possible to merge by dates within a range?

This problem is pretty straightforward in PROC SQL (and a real mess with a data step).

Look at the BETWEEN operator for comparison in the WHERE clause. You may need to do two steps, one for the exact matches and one for the inexact ones.
Frequent Contributor
Posts: 78

Re: Possible to merge by dates within a range?

Ok, that sounds like a great idea. I was looking into something called fuzzy merges and couldn't get it working exactly right, so I will give this a try.

Thanks!
Respected Advisor
Posts: 3,124

Re: Possible to merge by dates within a range?

Hi Doc,

quote:"and a real mess with a data step"

Well, not if hash() can be qualified as data step implementation.

Kindly Regards,

Haikuo

Contributor
Posts: 52

Re: Possible to merge by dates within a range?

I'd do something like this. If there is no chance of multiple matches, stick with new_dataset_ver1 and order it however you want.

If there is a chance that there might be more than one match per ID, look at the code used to obtain new_dataset_ver2. I calculate a variable that rates the closeness of the match. Here I take the distance in days as an absolute value, then select the match with the closest value. I use the monotonic() function which can be tricky if you do more than use it to add row numbers. Here it is an example of how to rank potential matches.

This is untested code, but I do use the technique with time stamped dates. Be careful to set up decision rules (in the order by clause) carefully. The code sample below could force an arbitrary selection between two matches with the same time distance.

proc sql;

   create table new_dataset_ver1 as

      select

         a.ID,

         a.examdate as examdate_a,

         b.examdate as examdate_b,

         b.examdate_p30,

         b.examdate_m30,

         abs(a.examdate - b.examdate) as difference

      from

         dataset1 as a,

         dataset2 as b

      where

         a.ID = b.ID and

         b.examdate_p30 < a.examdate < b.examdate_m30

      order by

         ID,

         difference;

        

   create table monotonic as

      select

         *,

         monotonic() as line_number

      from

         new_dataset_ver1;

  

   create table min_linenumber as

      select

         ID,

         min(line_number) as closest_date

      from

        monotonic
  group by
        ID;

  

   create table new_dataset_ver2 as

      select

         a.*

      from

         monotonic as a,

         min_linenumber as b

      where

         a.ID = b.ID and

         a.line_number = b.closest_date;        

quit;

Respected Advisor
Posts: 4,649

Re: Possible to merge by dates within a range?

You can try something like this (untested), it tries to deal properly with missing matches in dataset b :

proc sql;
create table temp as
select a.*, b.examdate as otherDate, b.var1, b.var2,
     abs(intck("DAY", a.examdate, b.examdate)) as sepDays
from a left join b on a.id = b.id
where calculated sepDays <= 30
order by a.id, a.examdate, sepDays;
quit;

data want(drop=sepDays);
set temp;
by id examdate;
if first.examdate;
run;

PG

PG
Ask a Question
Discussion stats
  • 5 replies
  • 4381 views
  • 0 likes
  • 5 in conversation