Well, I figured out the failure: I added my options to the top of the .cfg file, but they were being reset (to the lower values) at the bottom of the file! The hash version is running now, and clearly using nearly all the RAM.
For this kind of failures you the reason there is a usermods_config file specified. All usermodifications should go in that one, not in the default one supplied by SAS.
This is a bad habit seen by oldies of the V8/V6 time where the default installation did not support that. A technical guy could add that functionality.
With 9.3 you can included/nest config files and it setup that way.
Here are the timing results based on my real dataset:
SQL method: 5 hours
SET-BY method: 55 seconds :smileygrin:
SET-POINT method: untried (It seems to me to replicate SET-BY, but requires more coding)
HASH method: 2 hours (too few keys compared to overall data causes near-linear searching)
I learned some new tricks. Thanks to all who contributed ideas.
Thanks for the feedback! It is good to know that old-school data step still rocks if you know how to swing. Also, to my surprise, Proc Sort takes way less time than I expected.
Haikuo
Join us for SAS Innovate April 16-19 at the Aria in Las Vegas. Bring the team and save big with our group pricing for a limited time only.
Pre-conference courses and tutorials are filling up fast and are always a sellout. Register today to reserve your seat.
Learn how use the CAT functions in SAS to join values from multiple variables into a single value.
Find more tutorials on the SAS Users YouTube channel.