Solution Options: Finding Matches for Very large Data set using Data Flux Clustering

[Rama_V]

Hi everyone,

I have to find the matches based on matching rule for large data set of  ~40 million records and growing. I have designed to generate match codes using parallel processing(~4hrs for 60 million records for first time) and later generating match codes only for delta records in minutes. So,this reduced time significantly.

Using Clustering node, I can achieve matches but it is taking way too much time like 3-4 hours.

Any suggestions to improve performance and reduce time.

Thanks,

Rama 

[VincentRejany · Accepted solution]

Hi Rama

Reclustering your entire set of records won't be efficient and the process duration will extend over time.

Once you have calculated matchcodes and clustered your initial dataset, the future updates could be added by using match codes and sql lookups which will contain the same rules as your clustering node.

Basically you try to find for your new records possible matches in your initial dataset.

• If you don't find any so add a new cluster
• If you find one corresponding cluster, add the new record to this cluster
• If you find more than one corresponding cluster, either you do evaluate the best match or you collapse these clusters into a single one (the older or the younger).

Hope that helps.

Vincent

[Rama_V]

Thanks Vincent,

I believe it will improve performance.