topic Re: Sample size n and significance in Statistical Procedures

Sample size n and significance

bncoxuk — Mon, 04 Jul 2011 18:14:33 GMT

Hi Sir,

I have a very large dataset (>500k records) and will use it to run linear regression. The large sample size makes the power of statistical tests also very large. This means that any tiny little effect will cause the null hypothesis H0 to be rejected. It becomes very impractical to judge the significance of a variable simply based on p-value.

Is there any reference or rule which provides a guideline for the sample size and significance level alpha? For example, rather than treat 5% as significance level for small sample, do I use 0.1% as significance level? So if the p-value for a variable is 2%, it is treated as not significant.

Thanks for your clarification and suggestion.

Sample size n and significance

Ksharp — Tue, 05 Jul 2011 02:51:39 GMT

I recommend you to use Confidence interval .

Ksharp

Sample size n and significance

Reeza — Tue, 05 Jul 2011 04:51:27 GMT

Confidence interval and cut offs are related so essentially the same test.

I recommend a multistep model building approach.

1. Hold back a set of data, ie use 250k for model fitting and 250k as model testing

2. Use a cutoff of 0.025 or less to test if a variable is significant

3. Test in the hold back dataset for significance.

4. Repeat with different hold back samples to ensure there is actually an effect. You'll also want to differentiate between statistically significant results and practically significant.

Re: Sample size n and significance

Ruth — Tue, 05 Jul 2011 09:41:15 GMT

Hi, Ksharp, and Reeza, I also want to understand this bit. Quite good knowledge.

Can I ask why confidence interval is a good option when the sample size is large? Confidence interval only tells if it contains the point of zero. If the sample size is large, then the confidence interval should be very narrow. Any more information from confidence interval that i don't know?

Re: Sample size n and significance

DLing — Tue, 05 Jul 2011 14:45:51 GMT

You are right that as sample size increases, the ability to resolve increases, and you are able to split hair if you have huge data volume. Regression with multi-million records are being done routinely, something that's just not imaginable not that long ago. At some point one need to stop asking only statistical significance (is it really there or not?) and start asking context significance (so that 0.00001% difference is truly there, so what?). Ruth's point #4 last sentence is absolutely bang-on.