BookmarkSubscribeRSS Feed
Kurt_Bremser
Super User

How long is client_id defined? And what is your MEMSIZE option?

A quick calculation shows me that your ~33 million observations should fit in about 1.5 GB of RAM, unless the variable is longer than expected.

ChrisNZ
Tourmaline | Level 20
Kurt, hash tables have a significant per-record overhead. Look for macro hashsize or hash_size in these pages. Sorry can't look from my phone.
sfffdg
Obsidian | Level 7

Hi Kurt,

 

The Id is 10chars

current MEMSIZE=2147483648

ChrisNZ
Tourmaline | Level 20

Try adding line

  dcc.definedata("_N_");

to reduce the memory footprint,

 

sfffdg
Obsidian | Level 7
Thanks Chris,
Stillthe same issue
ChrisNZ
Tourmaline | Level 20

> Stillthe same issue

Then your data is too large to fit in memory. Your options are:

- Increase MEMSIZE. That's a startup option.

- Use the existing code. It works.

- Load successive subsets of the lookup table in the hash table and do the filtering in several stages (probably several data steps with a different portion of the look up table)

- Merge the tables. You need them to be sorted to do this efficiently. 

 

 

ChrisNZ
Tourmaline | Level 20

> Can any one help with this.

You now know what the code does. Wasn't that your question?

hackathon24-white-horiz.png

The 2025 SAS Hackathon has begun!

It's finally time to hack! Remember to visit the SAS Hacker's Hub regularly for news and updates.

Latest Updates

How to Concatenate Values

Learn how use the CAT functions in SAS to join values from multiple variables into a single value.

Find more tutorials on the SAS Users YouTube channel.

SAS Training: Just a Click Away

 Ready to level-up your skills? Choose your own adventure.

Browse our catalog!

Discussion stats
  • 21 replies
  • 2827 views
  • 2 likes
  • 4 in conversation