13 hours? If you have to do this again, I'd strongly suggest using the array solution, which should be noticably faster. In the program code below processing 100,000,000 skinny observaions, the T2 dataset (using arrays) took about half the time to create as the T1 (using hash). About 84 seconds vs 44 on our server.
hash is more flexible, but when it comes to looking up single values, it is intrinsically slower than arrays, since the latter has a key pointing directly to the array location of data value, whereas hash needs to process the key value before locating the data value.
... View more