topic Re: How do you unit test your DI jobs? in SAS Data Management

How do you unit test your DI jobs?

HeG — Wed, 26 Sep 2018 13:33:40 GMT

Hi,

We have DI jobs that we create.

In one DI job we have lots of tasks done.

How do you unit test your DI jobs?

If we write Macros, it is easy to have modules that we unit test. Different story with DI jobs.

//Henrik

Re: How do you unit test your DI jobs?

ClarkLawson — Thu, 04 Oct 2018 07:45:38 GMT

For our DI jobs we do the following:

1. Test that it runs successful in DI.

2. If you are using a batch scheduler, test it within the scheduler. For flows, this will mean running the whole flow.

3. If that flow is part of a larger module, then we would deploy the flows into a test environment and do a full end to end test.

Most of the above is manual.

Does that help?

Re: How do you unit test your DI jobs?

HeG — Thu, 04 Oct 2018 08:00:23 GMT

Hi,

Thank you for your answer.

Step 1 is maybe some sort of Unit Test. The question is what is the smallest "unit" I can test. For functional approaches we might consider a single function as a unit. But this is not to simple to test with DI jobs

Step2 and Step3 are more for Integration testing me.

Integration testing consists of executing groups of units in order to test their interaction and to see if they behave as predicted by the Design Specification. Units should only be integration tested once they have successfully been unit tested. Integration tests can be planned in a layered fashion so that they test increasingly large groups of units until the full system is constructed and tested.

Re: How do you unit test your DI jobs?

ClarkLawson — Thu, 04 Oct 2018 09:12:40 GMT

For 1, we run the full DI job rather than partially. Mainly because apart from our source(s) and target tables being registered, the interim datasets are work tables and so would need the DI job to be run fully to create them.

Re: How do you unit test your DI jobs?

Patrick — Mon, 08 Oct 2018 02:34:10 GMT

I normally treat a job as the unit so that's what I would be testing.

As the developer I'd run the job step by step out of DIS and ensure every single node returns what I'm expecting. That's informal testing during development. For unit testing I'd execute the job multiple times testing for example situations like: source table with no rows, re-running job with same data, source table doesn't exist, target table doesn't exist, ....and so on. This can be out of DIS or via batch.

I'd sure have at least one run in batch as this is how things need to work in the end.