topic Re: Solution Options: Finding Matches for Very large Data set using Data Flux Clustering in SAS Data Management

Solution Options: Finding Matches for Very large Data set using Data Flux Clustering

Rama_V — Fri, 30 Aug 2019 04:40:31 GMT

Hi everyone,

I have to find the matches based on matching rule for large data set of ~40 million records and growing. I have designed to generate match codes using parallel processing(~4hrs for 60 million records for first time) and later generating match codes only for delta records in minutes. So,this reduced time significantly.

Using Clustering node, I can achieve matches but it is taking way too much time like 3-4 hours.

Any suggestions to improve performance and reduce time.

Thanks,

Rama

Re: Solution Options: Finding Matches for Very large Data set using Data Flux Clustering

VincentRejany — Fri, 30 Aug 2019 08:01:29 GMT

Hi Rama

Reclustering your entire set of records won't be efficient and the process duration will extend over time.

Once you have calculated matchcodes and clustered your initial dataset, the future updates could be added by using match codes and sql lookups which will contain the same rules as your clustering node.

Basically you try to find for your new records possible matches in your initial dataset.

If you don't find any so add a new cluster
If you find one corresponding cluster, add the new record to this cluster
If you find more than one corresponding cluster, either you do evaluate the best match or you collapse these clusters into a single one (the older or the younger).

Hope that helps.

Vincent

Re: Solution Options: Finding Matches for Very large Data set using Data Flux Clustering

Rama_V — Sun, 01 Sep 2019 23:17:50 GMT

Thanks Vincent,

I believe it will improve performance.