Abhigyan Tracker

Re: Performance Management/Fine tuning of DataFlux Jobs

Abhigyan — Fri, 17 Jan 2014 07:30:37 GMT

Thanks Shawn. That is a good point you have raised. I will come back to you by implementing the changes.

Re: Performance Management/Fine tuning of DataFlux Jobs

Abhigyan — Thu, 16 Jan 2014 14:26:29 GMT

Hi Shawn

Will definitely try increasing the branch memory cache size option. Also will change the commit frequency and check.

BTW, for our job, we are writing to the DB using an expression where we are using the Expression Engine Language and DSNs to write data.

Re: Performance Management/Fine tuning of DataFlux Jobs

Abhigyan — Thu, 16 Jan 2014 08:31:24 GMT

Hi Shawn

I have put that "Lookup Map..." in front of branch. Its executing a bit faster. Anything apart from that which can be implemented, such as changing any of the advanced properties or the configuration files??

Re: Performance Management/Fine tuning of DataFlux Jobs

Abhigyan — Wed, 15 Jan 2014 06:02:14 GMT

Hey Shawn

Thanks a lot for the reply.

Following the branch node, i am just adding a few columns and passing some values to those columns. In the "Add GAR.." expression node, I am adding a column. No filtering over there.

In the embedded jobs, I am using a data job to check for a certain pattern using a if else statements.

if(ascii(x)==49 or ascii(x)==50 . . . . . . or ascii(x)=57)

return true

else

return false

Performance Management/Fine tuning of DataFlux Jobs

Abhigyan — Tue, 14 Jan 2014 11:59:12 GMT

Hi there!!

I have created a complex job which branches out to around 10 flows. It starts with a SQL read node which is reading data from a SQL Server database. The table from which data is being read contains around 3.2 million records. A filter has been added and around 1.6 million records are being written. After reading, the data is being processed for quality.

All these is taking up a lot of time - approximately 20 minutes for reading the data and around 4 hours for processing the data and completing the job.

Is there any way that DataFlux can be configured to complete jobs/read and process data at a much faster way? Can any configurations be done in order to fine tune the jobs?

I have attached a screenshot to illustrate the time the job is taking.