Data Management Studio - how much RAM do you really need?

[NigelLight]

Hi peers!

I am writing and running jobs using SAS Dataflux (v2.4) within Data Management Studio. To date this has been on a Intel i5 64bit Windows7 machine equiped with 4Mb of RAM.

Occasionally, especially with large jobs, this would cause problems and I would be asked to close down the application or even that I had 'attempted to read/write protected memory and it had become corrupt'. This generally occurred whilst developing code. Additionally, often whilst running jobs, the machine was working hard - there appeared to be a lot of disk activity and from monitoring, it appeared that all available memory (just under 3Mb) was being used

To attempt to address this, I increased the memory in my machine to 8Mb. However, this had little effect on job times and investigating further, it appeared that Dataflux still only used just under 3Mb of RAM

Reading through the SAS literature - it advises "a minimum of 2Mb"

If anybody can add to the above experience and advise on the optimum pc setup I would be most grateful. I am beginning to think that Dataflux doesn't use the full 64bit capability?

Thanks

Nige

[jarno]

Hello Nige

As DM Studio is a 32bit Windows application, it is bound by the limitations of a 32bit process in terms of addressing memory, even though you have a 64bit OS with 8GB of memory. A single 32bit process can address less than 3GB of RAM at any given time.

The DM Servers on the other hand are 64bit software not bound by limitations of the 32bit world.

Rgds,

Jarno

[NigelLight]

Hi Jarno

Thanks for the response - we were beginning to suspect as much (though can't find this stated in the documentation anywhere)

Is there anything that we can do to make it more stable and avoid getting messages such as above when developing solutions? I'm guessing the increase in memory may assist in any contention with other applications?

Regards

Nigel

[bmedalen]

Hi Nigel,

We found a bug in DMP client v2.4 that when you zoom out to a very small size in Windows.  It will cause the DMP client to crash.  Is this similar to the issue you are experiencing with developing large jobs that cause your client to crash?

Thanks,

Brett

[NigelLight]

Hi Brett

The problem tends to especially occur in large jobs - when a change is made (and especially where several things have been changed) an error can occur informing me to close the application.

It appears that memory is being corrupted/overwritten and the application gets confused. There is normally no problem on reopening so it does sort itself out ok - just an inconvenience (and lost code) - I try to save regularly to reduce the impact of this.

Sounds similar?

Nige

[bmedalen]

Nigel,

The sounds like a similar characteristic in that the errors occur while working on a job with a large number of nodes.  We experienced the memory error crash while zoom out and it sounds like you experienced the memory error crash while doing normal development.   Might be a general memory buffer error in DMP v2.4 client when there is a job with a large number of nodes.

Thanks,

Brett

[NigelLight]

Hi Brett

It is definitely, based on our experiences, more prevalent in data jobs with a large number of nodes - any idea if this will be fixed in 2.5 (or any future patches)?

It also seems to be more prevalent when having a large number of other applications open

Tip of the day - close everything else down and keep data jobs small (though unfortunately this isn't always possible)

... and save regularly of course!

Thaks again for your help

Nigel

[MikeFrost]

Hi Nigel,

Jarno is correct about the limitations of DM Studio, which is currently 32-bit only (though I will note there are plans to make DM Studio a 64-bit application). As he notes, you can push jobs that you design in DM Studio out to a 64-bit DM Server and take advantage of a larger memory address space afforded to you there.

On the broader question of memory management in DM Studio, depending on what functionality you are using, there are various performance and memory tuning options available which could help you. For example, in Profile there are options that specify how much memory to set aside for caching etc. When running Data Jobs, some of the more memory intensive nodes have memory usage-related options that can be set using the Advanced Properties window. Examples include nodes that perform joins, sorts, clustering or address verification. Although the defaults normally work fine, you might want to experiment with these settings if you want to run more memory-intensive jobs on DM Studio rather than on DM Server. The caveat to allocating lots of memory to individual nodes in the same job is that you can over allocate the memory by accident. For example, if you give two nodes in the same data job each 600MB of memory and you only have 1GB available, the job will not complete because it can’t allocate the specified amount of memory. As a result, if you want to run a job that has a relatively large number of nodes, you may want to try adjusting some of these settings so that you don't bump against a limitation on DM Studio.

I hope this additional information is helpful,

Mike F.

[NigelLight]

Hi Mike

Thanks for the additional tips

Nige