Hi! You do not specify any details, so the only thing I can do is to give a fast "Coffe Break Answer":
Suppose you have 10 giga data values. Estimate the 99.9% quantile VERY approximately using a small sample oif values. Then read the values, note the number of values, take away every value that is smaller than the estimated 99.9% quantile value. (That value was perhaps "rather incorrect" , it was the 99% quantile).
Anyway you end up with 1% of 10 G, which is 10**8 values, or 100 M values. SInce you know the total number of values (the exact 10 G values), you also know that the smallest value in the 100 M data set, corresponds to 99.0 in the overall quantile calculation. So use the SLOT-method to calculate all the quantiles that you want. If the range of data values is VERY big, you can perhaps ask a computer specialist to write a small Assembler program. Look at a data value - if the power of 8 (or 16 on IBM computers) is too small, delete it, after counting it.
This method is a little more than one-pass. The Assembler routine is good to speed it up.
Do you follow me? Any questions? This is an easy problem, since you ONLY want the extreme qunatiles.
The problem is to calculated extreme quantiles AND also normal quantiles, together.
/Br Anders
... View more
