SAS Data Integration Studio, DataFlux Data Management Studio, SAS/ACCESS, SAS Data Loader for Hadoop and others

How do I collect job statistics in DIS 4.2?

Reply
Occasional Contributor
Posts: 5

How do I collect job statistics in DIS 4.2?

I would like to create 2 tables:
The first 1 I can do. I send the job status to a table, no issues there.
2: You know when a job completes in DI Studio the statistics tab is populated? Well, I want those results available in a table. In particular, I'd like to have 1 table where all my results are appended to ( I may run into an issue when jobs are run at the same time but I'll cross that bridge when I come to it). I want this table so that I may code some conditional logic around it and display the results in our Portal.

Thanks!!!
Super Contributor
Posts: 356

Re: How do I collect job statistics in DIS 4.2?

Another option is parse the log to get this information, as that is where the stats come from....

Create a job at the end of all your jobs to collect this information to populate a data set, you then wont have the issues about multiple access.

Barry
Occasional Contributor
Posts: 5

Re: How do I collect job statistics in DIS 4.2?

Posted in reply to twocanbazza
So is there an easy way to parse the log file? I saw "some" SAS examples where they help you parse general logs but I would much rather let DI Studios and ARM handle this for me. Then, I would register the output table in metadata and be on my way.
SAS Employee RMP
SAS Employee
Posts: 52

Re: How do I collect job statistics in DIS 4.2?

please have a look at this.

http://www.youtube.com/watch?v=sHTce8qgxsU&feature=player_embedded#at=351

A SAS partner (WWW.EOM.NL) has developed this to extract all relevent information from the SAS logs.
SAS Employee
Posts: 38

Re: How do I collect job statistics in DIS 4.2?

Two additional comments/options:

1. There was a excellent presentation at SGF2011 from the same SAS Partner that further describes details shown in the video in the previous posting. You can find the paper here: http://support.sas.com/resources/papers/proceedings11/138-2011.pdf

2. DI Studio job statistics in 9.2 use ARM technology which is an industry standard technique for capturing performance information. You can configure additional appenders for ARM to write the DI Studio performance metrics and other content out to tables such as database tables. There is information on how to configure ARM appenders and details about the ARM infrastructure available in SAS 9.2 here: http://support.sas.com/documentation/cdl/en/armref/61526/HTML/default/viewer.htm#titlepage.htm
Contributor
Posts: 66

Re: How do I collect job statistics in DIS 4.2?

If you are using Platform Process Manager you can also parse the logs from this. It will include the job and flow, return code, metadata id of the job, start and end times and cpu time.
Ask a Question
Discussion stats
  • 5 replies
  • 860 views
  • 0 likes
  • 5 in conversation