topic DataFlux - How to archive profiling reports in SAS Data Management

DataFlux - How to archive profiling reports

smicha — Thu, 24 Oct 2019 09:09:31 GMT

Hello Community,

I have read in the link below, about archiving profiling reports. How can this be achieved:

http://support.sas.com/documentation/onlinedoc/dfdmstudio/2.7/dmpdmsug/Content/dfU_T_Profile_performance.html

I use Microsoft SQL Server as the data storage for my repository.

Thank you in advance,

smicha

Re: DataFlux - How to archive profiling reports

Rama_V — Sun, 27 Oct 2019 00:00:36 GMT

I hope this blog should give you some ideas

https://blogs.sas.com/content/sgf/2016/02/11/share-dataflux-data-quality-profiling-metrics-in-sas-visual-analytics/

Once you extracted the results stored elsewhere, you can clean the repository tables, especially for profiling related tables.

Re: DataFlux - How to archive profiling reports

afroditiz — Tue, 31 Mar 2020 10:05:51 GMT

@Rama_V

In continuance of the thread, is there an exhaustive list of profiling-related tables stored in the repository, so that someone proceeds to cleaning them? Also, by cleaning you mean deleting or truncating those tables?

Because I have a case where a FRQDSTxxx table never got created for some reason (FRQDSTxxx-1 & FRQDSTxxx+1 are created), and i would like to examine the possibility "cleaning" the whole repository for profiling only.

Plus, does deleting the profile history inside DMS also delete the FRQDSTxxx tables or not?

Thank you very much

Re: DataFlux - How to archive profiling reports

Rama_V — Tue, 07 Apr 2020 01:02:29 GMT

You need to look at the Repository schema, accessing it shown in the blog link my previous reply. Rule executions, pattern distributions etc stores row logs which takes lot of repository space .

I am not sure this is recommended approach for cleaning. I never truncated the DMS tables. If you do need truncate then I am not sure how the relationships in the repository tables will behave. So need some caution around that especially folders,profile jobs, business rules etc.

If I am deleting and recreating the repository then I usually export the jobs, business rules, profile jobs and all the results are extracted and loaded into a Data mart. This way I have the results ,Jobs and business rules which will be used to recreate my repository. If using RDBMS as repository then its even better to use scripts to create the repository. The import the exported objects back into repository.

Re: DataFlux - How to archive profiling reports

VincentRejany — Wed, 08 Apr 2020 07:48:26 GMT

Find in attachment the process job mentioned in the blog. It allows to extract data profiling metrics from a DataFlux repository (file or db).

Hope that helps

Vincent

Re: DataFlux - How to archive profiling reports

afroditiz — Wed, 08 Apr 2020 08:00:04 GMT

Hi @VincentRejany thank you very much for your replies.

It seems that the problem I am facing was irrelevant with FRQDST dynamic construction logic, but it's good to know more information about cleaning.

I can't see the attachment, but this is the job I am using as well based on the link you have sent 🙂

Regarding the repository "migration" part as mentioned in your previous reply, I guess that, having the repository in RDBMS, in case of a new environment I would just have to export/import profiles, rules etc. and define the repository definition to 'look' at the RDBMS, so I imaginge no further action would be required for profiling tables.

Re: DataFlux - How to archive profiling reports

VincentRejany — Tue, 14 Apr 2020 09:28:42 GMT

Strange that the file has been removed

Re: DataFlux - How to archive profiling reports

VincentRejany — Tue, 14 Apr 2020 09:29:43 GMT

And yes you can import export data profiling jobs (not reports), rules from on repository to another (file or db)