SAS Index Merge

[Kurt_Bremser]

How long is client_id defined? And what is your MEMSIZE option?

A quick calculation shows me that your ~33 million observations should fit in about 1.5 GB of RAM, unless the variable is longer than expected.

[ChrisNZ]

Kurt, hash tables have a significant per-record overhead. Look for macro hashsize or hash_size in these pages. Sorry can't look from my phone.

[sfffdg]

Hi Kurt,

The Id is 10chars

current MEMSIZE=2147483648

[ChrisNZ]

Try adding line

  dcc.definedata("_N_");

to reduce the memory footprint,

[sfffdg]

Thanks Chris,
Stillthe same issue

[ChrisNZ]

> Stillthe same issue

Then your data is too large to fit in memory. Your options are:

- Increase MEMSIZE. That's a startup option.

- Use the existing code. It works.

- Load successive subsets of the lookup table in the hash table and do the filtering in several stages (probably several data steps with a different portion of the look up table)

- Merge the tables. You need them to be sorted to do this efficiently. 

[ChrisNZ]

> Can any one help with this.

You now know what the code does. Wasn't that your question?