BookmarkSubscribeRSS Feed

Uniform Hashing of Arbitrary Input Into Key-Exclusive Segments

Started ‎04-19-2021 by
Modified ‎04-26-2021 by
Views 1,251
Paper 1032-2021
Authors

Paul Dorfman, Don Henderson

Abstract

Aggregating or combining large data volumes can challenge computing resources. For example, the process may be hindered by the system limits on utility space or memory and, as a result, either fail or run too long to be useful. It is a natural inclination to try solving the problem by segregating the input records into a number of smaller segments, processing them independently and combining the results. However, in order for such a divide-and-conquer tactic to work, two seemingly contradictory criteria must be met: First, to aggregate or combine the data correctly, no segment can share its key values with the rest; and second, the segments must be more or less equal in size. In this presentation, we show how a hash function can be used to achieve it for arbitrary input with no prior knowledge of the distribution of the key values among its records. Effectively, the method renders any task of aggregating or combining data of any size doable by splitting its input into a large enough number of segments. Such an approach can be used to process the segments sequentially or in-parallel. The trade-off is the need to partially re-read the data. However, it is a rather small price to pay for making a failing or endlessly running task finish on time.

Watch the presentation

Watch the authors present this topic -- Uniform Hashing of Arbitrary Input Into Key-Exclusive Segments -- on the SAS Users YouTube channel.

 

Comments

@DonH

@hashman 

 

Don, Paul, 

 

Thank you for this great article! Master-peace as usual!

 

Bart

Version history
Last update:
‎04-26-2021 03:50 PM
Updated by:
Contributors

SAS INNOVATE 2024

Innovate_SAS_Blue.png

Registration is open! SAS is returning to Vegas for an AI and analytics experience like no other! Whether you're an executive, manager, end user or SAS partner, SAS Innovate is designed for everyone on your team. Register for just $495 by 12/31/2023.

If you are interested in speaking, there is still time to submit a session idea. More details are posted on the website. 

Register now!

Get the $99 certification deal.jpg

 

 

Back in the Classroom!

Select SAS Training centers are offering in-person courses. View upcoming courses for:

View all other training opportunities.

Article Tags