topic Re: Hash object memory getting full in SAS Programming

Hash object memory getting full

navin1984 — Thu, 05 May 2022 21:45:46 GMT

Hi,
Data increased 2 times and i am using HASH object to look up the data. SAS job is failing due to insufficient memory for HASH object. I checked MEMSIZE, it is 2 GB while in hash object i have 11 million records.

1. Is there way to calculate the size of hash object at run time?
2. Is there any way to increase the size of hash object at run time?

Re: Hash object memory getting full

SASKiwi — Thu, 05 May 2022 22:22:42 GMT

A MEMSIZE of 2GB is a bit low these days. To give you some idea our current SAS server MEMSIZE setting is 6GB. Talk to your SAS administrator about an increased limit. At the end of the day hashes are loaded into memory so there is a limit to their size so they are no suited to very large data volumes.

Re: Hash object memory getting full

andreas_lds — Fri, 06 May 2022 05:20:14 GMT

If you are using 32 bit SAS, then 2 GB is the maximum you can use. But your sas admin ~~should~~ must know this.

Re: Hash object memory getting full

KachiM — Fri, 06 May 2022 05:50:37 GMT

There are ways to run programs with limited resources. Consult Paul Dorfman's papers. If you explain your issue with a constructed SAS dataset and your requirements, someone can come out with an efficient way to tackle memory issues.

Re: Hash object memory getting full

Ksharp — Fri, 06 May 2022 12:05:36 GMT

1) If you are running Batch way ,try add option " -memsize 0 " to maximize the memory sas used.
2) If You can edit " autoexec.sas " ,try add option " -memsize 0 " to maximize the memory sas used.

Re: Hash object memory getting full

Quentin — Fri, 06 May 2022 12:20:06 GMT

Just want to second the suggestion for reviewing Paul Dorfman's papers (and book) on hashing.

Think about limiting the variables in the hash table (do you need them all?). As a sanity check, if you load only the keys, will they fit in memory?

Sometimes you can hash in chunks (so when processing a dataset, load a hash table with data from a subset of rows, use the hash table, and then empty the hash table before loading it with data from the next chunk).

If you can't do what you want with hash tables, there are of course other approaches for lookups that are less memory intensive but may be slower / more disk IO.