Hash object memory getting full

[navin1984]

Hi,
Data increased 2 times and i am using HASH object to look up the data. SAS job is failing due to insufficient memory for HASH object. I checked MEMSIZE, it is 2 GB while in hash object i have 11 million records.

1. Is there way to calculate the size of hash object at run time?
2. Is there any way to increase the size of hash object at run time?

[SASKiwi]

A MEMSIZE of 2GB is a bit low these days. To give you some idea our current SAS server MEMSIZE setting is 6GB. Talk to your SAS administrator about an increased limit. At the end of the day hashes are loaded into memory so there is a limit to their size so they are no suited to very large data volumes. 

[andreas_lds]

If you are using 32 bit SAS, then 2 GB is the maximum you can use. But your sas admin should must know this.

[KachiM]

There are ways to run programs with limited resources. Consult Paul Dorfman's papers. If you explain your issue with a constructed SAS dataset and your requirements, someone can come out with an efficient way to tackle memory issues. 

[Ksharp]

1) If you are running Batch way ,try add option " -memsize 0 " to maximize the memory sas used.
2) If You can edit " autoexec.sas " ,try add option " -memsize 0 " to maximize the memory sas used.

[Quentin]

Just want to second the suggestion for reviewing Paul Dorfman's papers (and book) on hashing.

Think about limiting the variables in the hash table (do you need them all?).  As a sanity check, if you load only the keys, will they fit in memory?

Sometimes you can hash in chunks (so when processing a dataset, load a hash table with data from a subset of rows, use the hash table, and then empty the hash table before loading it with data from the next chunk).

If you can't do what you want with hash tables, there are of course other approaches for lookups that are less memory intensive but may be slower / more disk IO.