bluefusion/dataflux error - out of memory / overflow

[shawndylan]

Hi,

I just started using SAS DM for a week plus and having alot of errors due to memory. Can someone please help me please please please?

I have around 10 separate data sources but with very limited data (less than 50 records each). After i union and did standardisaion on the selected field, i wanted to perform a data matching. but i keep getting the following errors: either " [64:MATCHCODE:Match Codes 2] BlueFusion Plugin - DQ Engine error -600: BlueFusion - Out of memory." or "saying the memory overflow". 

no settings have been done to app.config because i do not know how to change it. 

And under the option for memory allocation for use during sorting and clustering options, they are set at 512MB.

I also have a cluster task after the data match with chucksize default setting at 16777216.

Is there any resolution on this please? Thanks.

[RonAgresta]

Hi,

If you haven't tried setting your sort and clustering nodes to use a lot less memory, you should. If I remember correctly, the processing engine will allocate memory for those at the start of your job and it could mean there's not enough memory to either load the Quality Knowledge Base or use the data quality functions. If there are just 50 records in each source, the sort and cluster memory values can be a lot smaller than 512MB. Try starting with 16MB each.

Ron