12-19-2015 12:58 PM
I loading a data from csv to SAS using SAS DI. Initially table contains the 16 variables, but now suddenly source contains 17 variables so I have added the same variable to source and target properties and updated the same. It is perfectly showing in DI but in EG table it is showing only 16 variables, it is not updated in EG. Now I have to add the new variable at 8th column position. Anybody have any idea How can we do that. Below is the example:
Old table Structure:
ID Name Order_NO unit_price
New Table structure:
ID Name Order_ID(New _column) Order_No unit_price
Any advise and help is appericiated....
12-21-2015 04:33 AM
Do you have an import agreement in place? If not then that is your problem. The ideal process is to first, before any data is sent, agree with the vendor what the file format will be, what it will contain. Write this down in a document and have both parties sign this. This forms the basis of what they should send you, and what you should receive. If the data received is not per that agreement, you go back to vendor and ask them to provide another file. As with anything programming -> Define, document, test, proceed to production.
As for your problem, its likely that on the first import your dataset was setup, and there is data there. Now it doesn't want to change that structure. You would need to (and I don't use DI, so don't know) change the underlying dataset, either through code, or in your table manager so that column is present, then reimport the data. Again, this is where that agreement is vital as changes to the data require changes to the database.
12-21-2015 11:04 AM
Even if @RW9 is not a DI Studio user, he's on to it.
DI Studio is a metadata specification tool, so it lacks a bit of the data management parts, like automatically create alter table statements.
So, you add columns in metadata, create mappings from the source.
If your target table have's a some kind of update strategy in the table loader, chances are that you will receive an error when redploying your job code.
So you need to add new/changed columns to the physical table in this scenario. If you are working with SAS tables/data sets - I'm sorry, but a rewrite of all data will occur.
If you are/or want to become a mature data warehouse shop, you need to implement these kind of actions in your production/maintenance process.