Maximum Exceeded Idle Time in Data Management Studio

[Shakti_Sourav]

Dear Team,

I am getting Maximum Exceeded Idle Time Issue in Data Source Node in Data Management Server after executing 2 crores rows. 

We have total 7 crores rows in Data Source Node. 

Please suggest me how to resolve it ? I have attached Screenshot for your references.

Target Node Options : for performance tuning 

Attached Error Screenshot :

attached Data Connection in SAS Dataflux ;

1.

2.  How to Increase performance.

3.

Please find it the Screenshot and let me know if required any changes.

and How to increase performance to read Data from database. It's taking 9 hours to read the table which contains 7 crores records.

Thank You Shakti 

[SASKiwi]

I suggest you try changing the commit interval. By using the "Commit all rows in a single transaction" option you are telling DataFlux to wait until all rows are processed which is obviously exceeding the database connection timeout limit. Try selecting "Commit every nnn rows" and maybe 5,000 or 10,000 rows. You may need to experiment with different values to get optimal performance and also stay within the timeout limit.

EDIT: I may have misinterpreted your screenshot. So you are doing bulk inserts, in batches of 50,000? Maybe stay with that option but use 25,000 or 10,000? Does that avoid the timeout?

[Shakti_Sourav]

Okay, but i got error in Data Source1, I have attached screenshot please find it and let me know what changes required.

[SASKiwi]

That's a different problem related to reading data. Is Data Source 1 reading from Oracle? If so check your data connection and DSN definition. Try a Data Preview just on that task. Does it work? Without more details of any errors I have no idea what the problem is. 

[Shakti_Sourav]

Yes, Data Source 1 is reading from ORACLE. Data Source having 7 crores rows. after 2 crores read i got Maximum exceeded idle time error. 

If i am taking small rows like 1 lakhs or 2 lakhs, it's working fine. when i am taking 7 crores. then we got this error. 

I have attached Job diagram Screenshot and log details. Please find it and let me know if required any other details

1. Job Diagram Screenshot,

2.  Log details,

3. data Preview is working,

1. it's taking 7 to 8 hours to read the data, is there any other way to reduce the reading time ? 

2. Batch job taking 28 hours or above to complete, i want to execute the batch job within 15 hours. is it possible ? if yes, please suggest how to do ? I have tested in parallel using the Fork Node, but it's taking almost same execution time. If need any details regarding the job details or data reading details, please let me know. It's needful to complete.