BookmarkSubscribeRSS Feed
milts
Pyrite | Level 9
Hi All,

Does anyone know how much memory needs to be set for memsize when i need to create a hash object whose source is a data set containing more than 32M records? Isn't it that the max size of hash object depends on the memory a machine has? Please correct me if I am wrong on this one.

I encountered a memory error regarding this. Currently memsize set in sasv9.cfg is 2G.

Help is much appreciated.

Thanks!
Milton
4 REPLIES 4
Ksharp
Super User
Hi.
>Isn't it that the max size of hash object depends on the memory a machine has?

I am not sure whether The size of hash object is depend on the memory.But you can expand buckets of hash table to speed, The max size of hash table buckets is 2^16,
You can specify " hashexp: 16 " to maximize which can promote the speed.


Ksharp

Message was edited by: Ksharp
Peter_C
Rhodochrosite | Level 12
have a look at usage note
34193 - How to determine how much memory my hash table will require
http://support.sas.com/kb/34/193.html
This Note is about the data you will put in the hash, rather than the environment. If 2G is not enough for your hash, you can set memsize to 0 to make all memory available.
the "fine" manual page to read is at http://support.sas.com/documentation/cdl/en/hostwin/63285/HTML/default/viewer.htm#win-sysop-memsize....

peterC
milts
Pyrite | Level 9
Thanks KSharp and PeterC for your responses.

I'll be loading a dataset with 32 million records into a hash object. one key and one data. please see below if my parameters for hash_test

%hash_test(rows=32000000,keycnt=1,
k1len=8,k1typ=n,
datacnt=1,d1len=8,d1typ=n
) ; run;

I tried running the macro hash_test and got the ff stats:

XMLRMEM available = 610,404,352

Actual table size: 1,024,000,000 bytes
Row size: 32 bytes
Optimal HASHEXP: 25


Given an optimal hashexp of 25, does that mean that it actually won't fit the hash object since max hashexp is 16?

Please correct me if I am wrong.

Thanks!
Ksharp
Super User
>Given an optimal hashexp of 25, does that mean that it actually won't fit the hash object since max hashexp is 16?


Yes.Your "hashexp of 25" is identified with 16,because hash table's max buckets is 2^16.
From your file's size is 1G, I think it is no problem to load your data with your computer.


Ksharp

sas-innovate-2024.png

Join us for SAS Innovate April 16-19 at the Aria in Las Vegas. Bring the team and save big with our group pricing for a limited time only.

Pre-conference courses and tutorials are filling up fast and are always a sellout. Register today to reserve your seat.

 

Register now!

How to Concatenate Values

Learn how use the CAT functions in SAS to join values from multiple variables into a single value.

Find more tutorials on the SAS Users YouTube channel.

Click image to register for webinarClick image to register for webinar

Classroom Training Available!

Select SAS Training centers are offering in-person courses. View upcoming courses for:

View all other training opportunities.

Discussion stats
  • 4 replies
  • 1975 views
  • 0 likes
  • 3 in conversation