topic Re: Which is best in performance Creating Match Codes in Data Flux Vs Base SAS code Deploy on Server in SAS Data Management

Which is best in performance Creating Match Codes in Data Flux Vs Base SAS code Deploy on Server

Rama_V — Fri, 05 Oct 2018 00:17:36 GMT

Hi Everyone,

In our organisation we have to create match codes for huge to data(million records) for clustering.

I can understand you can create match codes and clustering using Data Management Studio and schedule on DM Server.

Currently, the development has been done on SAS code(using DQmatch function) job and scheduled on the server.

I can't justify which one is best approach to improve performance on same server. Using SAS code is taking hours to run.

So I am just wondering has anyone experience similar situation or have any information around this.

Please any assistance would be really appreciated.

Thanks,

Rama

Re: Which is best in performance Creating Match Codes in Data Flux Vs Base SAS code Deploy on Server

Patrick — Mon, 08 Oct 2018 01:55:14 GMT

Match code generation is a very resource intensive process. It always uses DataFlux whether you call this functionality now out of SAS or directly out of a DF job so I don't believe going for a DF job will improve performance.

What you could do to improve end-to-end runtimes:

1. Set-up parallel jobs each creating match-codes for a sub-set of your source data

2. Design and implement delta processing so don't re-create all match codes every single time but only create match codes for new or changed records.

Re: Which is best in performance Creating Match Codes in Data Flux Vs Base SAS code Deploy on Server

Rama_V — Tue, 09 Oct 2018 02:19:55 GMT

Thanks Patrick.

I really appreciate your inputs.