topic Re: Data migration in SAS Programming

Data migration

HeatherNewton — Tue, 11 Apr 2023 23:38:33 GMT

I move sas data to DB2 wheee we have aded keys in table. Now when i load data into tables, duplicates based on keys were deleted. This makes my uat data having less volume compared to production data. How can I address this gap? Not adding keys to db2 table? Declare to users it is known gap? I dont kniw if there us impact to later calculation yet as we are still at data preparation and it is not possible to decide from reading code.

Anyone have experience around these?

Re: Data migration

SASKiwi — Wed, 12 Apr 2023 00:49:58 GMT

Surely you have a business rule for this? Is data with duplicate keys allowed or not allowed (the answer might be different depending on the table)? If it is allowed then your DB2 table definition is incorrect and needs to be corrected. If it is not allowed, then your current Production data is incorrect and your new DB2 data is correct.

Re: Data migration

HeatherNewton — Wed, 12 Apr 2023 00:55:02 GMT

Data architect designed that so keys allowed to minimize data unless he is not awared there could be impact which could be the case too. User not known yet and guess they only care about impact to data. Suppose no material sifference before and after migration.

Re: Data migration

SASKiwi — Wed, 12 Apr 2023 01:21:56 GMT

"Data architect designed that so keys allowed to minimize data" - this isn't a business rule, it's an architectural decsion. I suggest you raise the issue of a lack of a business rule with your project team. Until you decide what this rule is you don't know if your migrated data is correct or not.

Re: Data migration

Kurt_Bremser — Wed, 12 Apr 2023 05:22:49 GMT

If only observations (records) were removed where all values (variables,columns) were identical, then no information was lost, and you should ask yourself where the duplicates came from.

If this is not the case, then information WAS lost, and you have to determine if this information is crucial, and how to make sure the correct "duplicate" is kept if information can be discarded. If information is crucial, the key architecture needs to be reviewed.