Cluster Init Error Occurs on DataFlux Server but not in Data Management Studio 2.4

[csdunn1us]

I have a Dataflux job that was developed in Data Management Studio 2.4 that contains a clustering step. In testing this job with either 1,000 or 100,000 records, the job will run with success if it is run via Data Management Studio (32 bit). Once it is deployed to the server (64 bit) and run there, the job fails with the following error:

(DF001:0 - 66: Cluster init error: failed to allocate memory)

Here is some other log information that is written prior to the above error:

2013-11-11T13:32:29,806 INFO  [00001764] - ODBC op success with info: [01000] [Microsoft][ODBC SQL Server Driver][SQL Server]Changed database context to 'MSIWStaging'. (5701);[01000] [Microsoft][ODBC SQL Server Driver][SQL Server]Changed database context to 'MSIWStaging'. (5703)

2013-11-11T13:32:29,870 INFO  [00001764] - Node DF001 failed.

2013-11-11T13:32:29,870 ERROR [00001764] - Node DF001 error: 66: Cluster init error: failed to allocate memory

2013-11-11T13:32:29,870 INFO  [00001744] - Setting Execution State: STOPPING, Error message:(Process Flow - Job terminated due to error in one or more nodes. (DF001:0 - 66: Cluster init error: failed to allocate memory))

2013-11-11T13:32:29,871 INFO  [00001744] - Job terminated due to error in one or more nodes.

2013-11-11T13:32:29,871 INFO  [00001744] - Setting Execution State: ERROR, Error message:(Process Flow - Job terminated due to error in one or more nodes. (DF001:0 - 66: Cluster init error: failed to allocate memory))

2013-11-11T13:32:29,871 ERROR [00001744] - Job failed to finalize: Process Flow - Job terminated due to error in one or more nodes. (DF001:0 - 66: Cluster init error: failed to allocate memory)

Our group is fairly new to working with DataFlux jobs and we have tried different ways to troubleshoot this issue. So far, we don't really have any answers.

If you have any thoughts or recommendations on this, please share. I hope that I have provided sufficient information about this issue. If I haven't, please inform me of other supporting information you need.

Thank  you for your help.

csdunn1us

[MikeFrost · Accepted solution]

Hi,

The Cluster node lets you specify how much memory it uses before it starts spilling to disk. The more memory you specify the faster it will run, however specifying too much results in it running out of memory. Try checking the advanced setting on the node to see how much memory is specified. If that value is not present, check the value of CLUSTER/BYTES in the config file. If either value is high enough that it would run out of memory and this error would result, so you should try lowering this value to accommodate.

If that fails to work, then you should probably open a support case.

Thank you,
Mike F.

[csdunn1us]

Thank you for the feedback, this is very helpful.