topic Re: DI Studio SCD Type 1 table loader in SAS Programming

DI Studio SCD Type 1 table loader

qwererty — Fri, 31 Jan 2014 17:04:40 GMT

Hi,

I am trying to utilise the type 1 table loader so I can update a table with only changed dimensions. The table has 70k rows and 5 variables. I require 3 varibale s as my lookup to check for any change and then I have one target column that is changed. This is the first time I have used the a SCD and I am finding it can take up to 15 miutes to run with potentially only 3k records changing. Firstly is this normal? Secondly how can I optimise this time? I have experimented with different hash lookup sizes and no noticeable improvement is made.

Any ideas?

Thanks

Re: DI Studio SCD Type 1 table loader

LinusH — Sat, 01 Feb 2014 08:55:16 GMT

Sounds like a heck of time for so little records, independent of update technique.

And yes, there are stuff you can do to optimize the code.

But first things first. Locate in the log the step(s) that consumes the most resources, and try to start from there.

What aboit other processing - is this the only job that performs poor? Or coudl it be a system constraint?

Why are you using the Type 1 Loader? I seldom use it. Depending on your requirement you could use a standard Table Loader, or the Type 2 Loader.