When you could check the sizing on the OS I think it is about 6-10Gb. On today's machines that should be no problem anymore having som 32Gb of internal memory (server) or 8Gb (lap-top). A harddisk space of 100Gb does not bother anyone. More usual are the 500Gb or Tb sizes. SSD's of 200Gb becoming commercial approaches. Proc sort en sql are both multitreading they can use the multiple processors you probably have (4-8 on a laptop I5/I7) or the same or more on a server. No real difference and easy coding. The job should be done within an half-hour as guestimate. Suppose you would have problems you could think of using the hash functions when all duplicates could be hased in memory and while processsing the incoming data. This could be part of stream/signal processing, That is probably a whole different approach not what you are looking for.  ... View more