topic Maintain cluster id created in previous runs of a data flow in SAS Data Management

Maintain cluster id created in previous runs of a data flow

Arlan — Fri, 09 Sep 2016 16:55:25 GMT

I would like to be able to retain the cluster id assigned to customers in previous runs of a data flow in Data Managerment Studio 2.6.

I have handled this in previous versons of dataflux with the Cluster Update node in dfpower architect 8.1/8.2. So far I haven't identified if this is still an option in or replaced by a different node.

Re: Maintain cluster id created in previous runs of a data flow

audrey — Wed, 14 Sep 2016 06:12:02 GMT

Cluster Update node did not guarantee to preserve clusterIDs of already clustered data rows. Whether or not the existing clusterIDs would change depends on whether any of the new data rows match into multiple existing clusters.

Cluster Update node was deprecated, you can use Cluster node instead.

In you Clustering job, make sure the “old” records are passed into the Cluster node in identical order each time, followed by the new records. This will guarantee that the “old” records will retain their original cluster IDs, as long as those clusters are not joined by the new records being added.

Re: Maintain cluster id created in previous runs of a data flow

Arlan — Wed, 05 Oct 2016 14:35:38 GMT

Thanks Audrey

I was afraid that it had been depreciated. Realizing that it didn't guarantee preservation of the cluster ids, it at least maintained the existing cluster id for those existing customers who didn't have changes to their customer information.

The loading of the old records into the Cluster node should work if we were using static data but our customer information is dynamic. In this case corrected information may trigger a cluster between these old records. This would cause renumbering of clusters after that point.

Example for those that may explore this option. Old customer records with cluster id 100 and cluster id 150 have their information corrected and both are now assigned to the same cluster 100 on future Cluster Node runs. This would be OK but customers after 150 will be renumbered.

I can take this one step further and join it back to the original cluster run and maintain existing cluster ids. I am not under this requirement at this time and will forgo this extra step.