SAS Data Integration Studio, DataFlux Data Management Studio, SAS/ACCESS, SAS Data Loader for Hadoop and others

Profiling on dynamic data source

Reply
Frequent Learner
Posts: 1

Profiling on dynamic data source

Hi,

 

We are trying to run the profiling process we have build and an error occurred regarding a field that has been retired from the source. Before this change the profiling process was completed successfully as expected. 

 

Please note that our data source is a view table. Thus, changes are expected in the table structure such as new or retired fields.

 

Is there any way to run the same profiling when the dataset is changing dynamically without modifying the process?

 

Thanks,

Ioanna

SAS Employee
Posts: 8

Re: Profiling on dynamic data source

[ Edited ]
Posted in reply to IoannaKPMGcy

Ioanna,

 

I see clearly the scenario you are depicting.  Unfortunately, once a Data Plan has been saved and established it expects to be served with the same schema for each run.  The data sources and content can be dissimilar, as long as the schema matches the original data source. 

 

As transformations are associated with a Data Plan, the transformations are working against specific columns/data types. When your view undergoes schema change, any transformations that worked directly against a "now-gone" column would be come invalidated.  This may invalidate one transformation and have a cascading effect through others. 

 

Answering your question directly;  structural changes made in the schema of your view would require intervention on your part.

 

Quick thought: most difficulties occur when your view has dropped a column and a transformations in your Data Plan performs actions on that dropped column.  Conversely, If your view added a column, the Data Plan does not worry quite so much because the new column has no association with transformations in your Data Plan.   

 

Regards,

Daniel

 

Daniel Paner

Principal Product Manager

SAS Viya Data Management

 

 

 

 

Ask a Question
Discussion stats
  • 1 reply
  • 104 views
  • 0 likes
  • 2 in conversation