topic SAS DI Job tracking details in SAS Data Management

SAS DI Job tracking details

Babloo — Fri, 06 Sep 2019 10:01:30 GMT

I want to track the SAS job name which was executed, job start time, job end time, job folder location and job status after the execution of every job in SAS DI 4.9. Is there any table which automatically pulls this details for every job? If yes, I can pull those information via SAS code which I can place in 'post code' section

Objective is to get these details from SAS and insert it into the SQL table.

Re: SAS DI Job tracking details

JosvanderVelden — Fri, 06 Sep 2019 15:13:16 GMT

Have you seen the paper "Using SAS® Job Monitor to Analyze SAS® Data Integration
Studio Job Performance": https://www.sas.com/content/dam/SAS/support/en/sas-global-forum-proceedings/2019/3586-2019.pdf

Re: SAS DI Job tracking details

Babloo — Fri, 06 Sep 2019 16:13:44 GMT

Could you please tell me how can I achieve the objective only via DI?
Because I do not have access to other softwares like Environment manager,
Management console....

Re: SAS DI Job tracking details

Patrick — Sat, 07 Sep 2019 00:46:09 GMT

@Babloo

I've seen people implementing such stuff via a custom transformation which they just add at the start and beginning of a DIS job - or if you don't need an entry if a job fails then just at the end. I've seen other people doing it via macro calls just added as pre- and post-code to the DIS job.

Transformation Return Code Check generates SAS code which gives you a very good start to develop such custom code.

BUT: DIS jobs get scheduled and run in batch. Many schedulers maintain a repository where such execution information gets stored. I believe you only should go for a DIS only solution if the scheduler doesn't maintain the information you need. LSF for example got this information and you can query it and extract the info into a SAS table.

Because the scheduling process is the parent, your SAS job can terminate in any way and you still maintain all the execution information. For example if an abort statement gets executed somewhere in the middle of your job (i.e. by a lookup transformation) then the node at the end of the DIS job writing status information gets never executed - but the scheduler will have the information that the job terminated with return code n.

Re: SAS DI Job tracking details

Babloo — Sat, 07 Sep 2019 00:56:44 GMT

I'm trying to implement the things in precode and postcode with the use of
macros. Again how and where can I store the values of macros without
creating datasets. Once I have this information in SAS, I have to feed the
values in SQL table.

Also could you please tell me where can I find the transformation return
code in DI job?

Re: SAS DI Job tracking details

Patrick — Sun, 08 Sep 2019 03:11:19 GMT

@Babloo

Add DIS transformation Return Code Check to any DIS job, create a new action and select "Send Entry to a Dataset". The generated macro code under the code tab will give you a very good starting point to implement what you're after.

Or if what comes out of the box is enough for you then you can also use the DIS job level status handling. You get there just by right clicking on the job canvas.

You can also select an action of custom - and custom could then be a SAS macro where you're doing whatever you need.

Re: SAS DI Job tracking details

Babloo — Sun, 08 Sep 2019 05:06:44 GMT

I have to extract variable value from the DIS job folder and I would like
to know where the folder details are stored apart from the properties of
the job.

If it not stored any where apart from properties, how can I extract value
from that?

Re: SAS DI Job tracking details

Patrick — Sun, 08 Sep 2019 08:34:54 GMT

@Babloo

A) Your question is not clear

B) You really need to start and look into and understand DIS generated code to get a better grasp what's available to you.

If you don't see what you need in generated code then "play around" changing options and then revisit the generated code and check if that gives you the required information.

You could of course always query SAS metadata for additional metadata information. Given that a deployed job (.sas file) doesn't need to be in sync with its originating metadata or that for moving a job (.sas file) the originating metadata doesn't necessarily need to be migrated as well, I wouldn't consider such metadata queries as good idea.

Re: SAS DI Job tracking details

Babloo — Sun, 08 Sep 2019 08:44:44 GMT

Assume my jobs are located in
/var/data/life/myjob
/var/data/nonlife/myjob

Now I want to create one new variable called DOMAIN which should say which
domain DIS job belongs to. In my example life and nonlife is domain and so
the new variable DOMAIN should have those values.

My question is how to find values for new variable from the DIS job folder?

Re: SAS DI Job tracking details

Patrick — Sun, 08 Sep 2019 09:09:06 GMT

@Babloo

And my answer is that you should ask a new question as a new question, that you should inspect the DIS generated code to determine if there is already a macro variable created that has this information in it. If not then you would need to query SAS Metadata for additional SAS Metadata information. DIS generates the metadata job ID into your code (&jobID) so you've got a very good starting point for doing such a metadata query (but as said: job metadata does not need to be in sync with the deployed .sas file so....).