SAS Data Integration Studio, DataFlux Data Management Studio, SAS/ACCESS, SAS Data Loader for Hadoop and others

Maintain cluster id created in previous runs of a data flow

Accepted Solution Solved
Reply
Occasional Contributor
Posts: 8
Accepted Solution

Maintain cluster id created in previous runs of a data flow

I would like to be able to retain the cluster id assigned to customers in previous runs of a data flow in Data Managerment Studio 2.6.


I have handled this in previous versons of dataflux with the Cluster Update node in dfpower architect 8.1/8.2.  So far I haven't identified if this is still an option in or replaced by a different node.


Accepted Solutions
Solution
‎10-05-2016 10:36 AM
SAS Employee
Posts: 14

Re: Maintain cluster id created in previous runs of a data flow

Cluster Update node did not guarantee to preserve clusterIDs of already clustered data rows. Whether or not the existing clusterIDs would change depends on whether any of the new data rows match into multiple existing clusters.

 

Cluster Update node was deprecated, you can use Cluster node instead.

 

In you Clustering job, make sure the “old” records are passed into the Cluster node in identical order each time, followed by the new records. This will guarantee that the “old” records will retain their original cluster IDs, as long as those clusters are not joined by the new records being added.

 

 

View solution in original post


All Replies
Solution
‎10-05-2016 10:36 AM
SAS Employee
Posts: 14

Re: Maintain cluster id created in previous runs of a data flow

Cluster Update node did not guarantee to preserve clusterIDs of already clustered data rows. Whether or not the existing clusterIDs would change depends on whether any of the new data rows match into multiple existing clusters.

 

Cluster Update node was deprecated, you can use Cluster node instead.

 

In you Clustering job, make sure the “old” records are passed into the Cluster node in identical order each time, followed by the new records. This will guarantee that the “old” records will retain their original cluster IDs, as long as those clusters are not joined by the new records being added.

 

 

Occasional Contributor
Posts: 8

Re: Maintain cluster id created in previous runs of a data flow

Thanks Audrey
I was afraid that it had been depreciated.  Realizing that it didn't guarantee preservation of the cluster ids, it at least maintained the existing cluster id for those existing customers who didn't have changes to their customer information.  
 
The loading of the old records into the Cluster node should work if we were using static data but our customer information is dynamic.  In this case corrected information may trigger a cluster between these old records.  This would cause renumbering of clusters after that point.
 
Example for those that may explore this option.  Old customer records with cluster id 100 and cluster id 150 have their information corrected and both are now assigned to the same cluster 100 on future Cluster Node runs.  This would be OK but customers after 150 will be renumbered. 
I can take this one step further and join it back to the original cluster run and maintain existing cluster ids.  I am not under this requirement at this time and will forgo this extra step.
☑ This topic is SOLVED.

Need further help from the community? Please ask a new question.

Discussion stats
  • 2 replies
  • 285 views
  • 0 likes
  • 2 in conversation