How to set up automatic nightly full lineage for web infrastructure platform

[tmd1]

Hi all,

We are looking to expose the lineage of all of our SAS tables for viewing in the SAS Lineage Viewer web application.

We believe that the BI Lineage Scan on the DM Server only works over:

1. Reports

2. Information Maps

We have manually added a couple of tables to an Information Map and run a test BI Scan over it for export to the Web Infrastructure Platform.

I have found an excellent post in this community by Paul Homes of Metacoda with a tip to run BI Scans nightly at a high level in the Metadata tree.

This is exactly what we are looking for. We would then want the output from that nightly scan across all SAS tables to be automatically available in the web apps without manual operations.

We have no previous DataFlux experience in our team and seek expert guidance as to what might work best to achieve this.

Our initial thoughts are that it might be possible to:

1. automatically schedule a nightly report of a type that references all SAS tables that the BI Lineage scan could be scheduled to run over and exported to the Web Infrastructure Platform.

2. automatically schedule a nightly process to populate a complete Information Map of all SAS tables that the BI Lineage scan could be scheduled to run over and exported to the Web Infrastructure Platform.

Any tips and ideas from the community will be greatly appreciated.

Trevor

[ccaulkins91]

Did you try the SAS administration community at large? Was there any response? I was told that these jobs are highly interactive therefore involves a large SAS admin effort.

[Patrick]

The questions you have sound a bit "bigger" than what can be coverered in a Forum like the one here.

If you found already a paper from the expert you're really happy with, then consider to engage with @PaulHomes on a commercial level to get all the expert advice you seek.

It's in my experience really worth to get the design approach right from start as this will save a lot of headache and effort.

N.B:

I've made this proposition for a commercial engagement without any personal agenda and I won't have any personal gains out of it whatsoever.