BookmarkSubscribeRSS Feed
bncoxuk
Obsidian | Level 7

Hi Sir,

I have a very large dataset (>500k records) and will use it to run linear regression. The large sample size makes the power of statistical tests also very large. This means that any tiny little effect will cause the null hypothesis H0 to be rejected. It becomes very impractical to judge the significance of a variable simply based on p-value.

Is there any reference or rule which provides a guideline for the sample size and significance level alpha? For example, rather than treat 5% as significance level for small sample, do I use 0.1% as significance level? So if the p-value for a variable is 2%, it is treated as not significant.

Thanks for your clarification and suggestion.

4 REPLIES 4
Ksharp
Super User

I recommend you to use Confidence interval .

Ksharp

Reeza
Super User

Confidence interval and cut offs are related so essentially the same test.

I recommend a multistep model building approach.

1. Hold back a set of data, ie use 250k for model fitting and 250k as model testing

2. Use a cutoff of 0.025 or less to test if a variable is significant

3. Test in the hold back dataset for significance.

4. Repeat with different hold back samples to ensure there is actually an effect. You'll also want to differentiate between statistically significant results and practically significant.

Ruth
Fluorite | Level 6

Hi, Ksharp, and Reeza, I also want to understand this bit. Quite good knowledge.

Can I ask why confidence interval is a good option when the sample size is large? Confidence interval only tells if it contains the point of zero. If the sample size is large, then the confidence interval should be very narrow. Any more information from confidence interval that i don't know?

DLing
Obsidian | Level 7

You are right that as sample size increases, the ability to resolve increases, and you are able to split hair if you have huge data volume.  Regression with multi-million records are being done routinely, something that's just not imaginable not that long ago.  At some point one need to stop asking only statistical significance (is it really there or not?) and start asking context significance (so that 0.00001% difference is truly there, so what?).  Ruth's point #4 last sentence is absolutely bang-on.

sas-innovate-2024.png

Available on demand!

Missed SAS Innovate Las Vegas? Watch all the action for free! View the keynotes, general sessions and 22 breakouts on demand.

 

Register now!

What is ANOVA?

ANOVA, or Analysis Of Variance, is used to compare the averages or means of two or more populations to better understand how they differ. Watch this tutorial for more.

Find more tutorials on the SAS Users YouTube channel.

Discussion stats
  • 4 replies
  • 1524 views
  • 2 likes
  • 5 in conversation