topic Re: Memory needed for Hash object in SAS Programming

Memory needed for Hash object

milts — Mon, 07 Mar 2011 03:35:06 GMT

Hi All,

Does anyone know how much memory needs to be set for memsize when i need to create a hash object whose source is a data set containing more than 32M records? Isn't it that the max size of hash object depends on the memory a machine has? Please correct me if I am wrong on this one.

I encountered a memory error regarding this. Currently memsize set in sasv9.cfg is 2G.

Help is much appreciated.

Thanks!
Milton

Re: Memory needed for Hash object

Ksharp — Mon, 07 Mar 2011 04:51:59 GMT

Hi.
>Isn't it that the max size of hash object depends on the memory a machine has?

I am not sure whether The size of hash object is depend on the memory.But you can expand buckets of hash table to speed, The max size of hash table buckets is 2^16,
You can specify " hashexp: 16 " to maximize which can promote the speed.

Ksharp

Message was edited by: Ksharp

Re: Memory needed for Hash object

Peter_C — Mon, 07 Mar 2011 16:39:14 GMT

have a look at usage note
34193 - How to determine how much memory my hash table will require
http://support.sas.com/kb/34/193.html
This Note is about the data you will put in the hash, rather than the environment. If 2G is not enough for your hash, you can set memsize to 0 to make all memory available.
the "fine" manual page to read is at http://support.sas.com/documentation/cdl/en/hostwin/63285/HTML/default/viewer.htm#win-sysop-memsize.htm

peterC

Re: Memory needed for Hash object

milts — Thu, 10 Mar 2011 12:53:47 GMT

Thanks KSharp and PeterC for your responses.

I'll be loading a dataset with 32 million records into a hash object. one key and one data. please see below if my parameters for hash_test

%hash_test(rows=32000000,keycnt=1,
k1len=8,k1typ=n,
datacnt=1,d1len=8,d1typ=n
) ; run;

I tried running the macro hash_test and got the ff stats:

XMLRMEM available = 610,404,352

Actual table size: 1,024,000,000 bytes
Row size: 32 bytes
Optimal HASHEXP: 25

Given an optimal hashexp of 25, does that mean that it actually won't fit the hash object since max hashexp is 16?

Please correct me if I am wrong.

Thanks!

Re: Memory needed for Hash object

Ksharp — Fri, 11 Mar 2011 05:18:07 GMT

>Given an optimal hashexp of 25, does that mean that it actually won't fit the hash object since max hashexp is 16?

Yes.Your "hashexp of 25" is identified with 16,because hash table's max buckets is 2^16.
From your file's size is 1G, I think it is no problem to load your data with your computer.

Ksharp