"Duplicate values not allowed" on my pk_index in DI Studio

[EinarRoed]

I've created a job that's using a composite PK which includes eight variables. When I run the job, the final Table Loader generates an error message telling me  "Duplicate values not allowed on index pk_index for file MYTABLENAME." The Constraint- and Index Conditions of the Table Loader are both set to "After Load: On (table creation)". What is a good way to resolve this situation? I've never handled something like this before.

Thanks for your time.

[LinusH · Accepted solution]

Obviously, you have duplicate values in your source data.

How to handle this is a matter of process.

Perhaps you have the source data i some kind of staging table, which will make it easy to examine and determine the duplicates.

If duplicates are common, you could add some logic in your flows to find the duplicates prior to the loading step, hand have them stored in a separate table. The Data validation transformation could be used for this.

About the constraints, what is the requirement to handle data quality? Having constraints on a target like this is like a guarantee that the data is consistent. You pay with a little overhead during load. The trade off is when to restore or in other ways clean up the target table after a load of duplicate values.