Profiling on dynamic data source

[IoannaKPMGcy]

Hi,

We are trying to run the profiling process we have build and an error occurred regarding a field that has been retired from the source. Before this change the profiling process was completed successfully as expected. 

Please note that our data source is a view table. Thus, changes are expected in the table structure such as new or retired fields.

Is there any way to run the same profiling when the dataset is changing dynamically without modifying the process?

Thanks,

Ioanna

[SAS-Dan]

Ioanna,

I see clearly the scenario you are depicting.  Unfortunately, once a Data Plan has been saved and established it expects to be served with the same schema for each run.  The data sources and content can be dissimilar, as long as the schema matches the original data source. 

As transformations are associated with a Data Plan, the transformations are working against specific columns/data types. When your view undergoes schema change, any transformations that worked directly against a "now-gone" column would be come invalidated.  This may invalidate one transformation and have a cascading effect through others. 

Answering your question directly;  structural changes made in the schema of your view would require intervention on your part.

Quick thought: most difficulties occur when your view has dropped a column and a transformations in your Data Plan performs actions on that dropped column.  Conversely, If your view added a column, the Data Plan does not worry quite so much because the new column has no association with transformations in your Data Plan.   

Regards,

Daniel

Daniel Paner

Principal Product Manager

SAS Viya Data Management