Turn on suggestions

Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.

Showing results for

- Home
- /
- Analytics
- /
- Stat Procs
- /
- Sample size n and significance

Options

- RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page

- Mark as New
- Bookmark
- Subscribe
- Mute
- RSS Feed
- Permalink
- Report Inappropriate Content

Posted 07-04-2011 02:14 PM
(1523 views)

Hi Sir,

I have a very large dataset (>500k records) and will use it to run linear regression. The large sample size makes the power of statistical tests also very large. This means that any tiny little effect will cause the null hypothesis H0 to be rejected. It becomes very impractical to judge the significance of a variable simply based on p-value.

Is there any reference or rule which provides a guideline for the sample size and significance level alpha? For example, rather than treat 5% as significance level for small sample, do I use 0.1% as significance level? So if the p-value for a variable is 2%, it is treated as not significant.

Thanks for your clarification and suggestion.

4 REPLIES 4

- Mark as New
- Bookmark
- Subscribe
- Mute
- RSS Feed
- Permalink
- Report Inappropriate Content

I recommend you to use Confidence interval .

Ksharp

- Mark as New
- Bookmark
- Subscribe
- Mute
- RSS Feed
- Permalink
- Report Inappropriate Content

Confidence interval and cut offs are related so essentially the same test.

I recommend a multistep model building approach.

1. Hold back a set of data, ie use 250k for model fitting and 250k as model testing

2. Use a cutoff of 0.025 or less to test if a variable is significant

3. Test in the hold back dataset for significance.

4. Repeat with different hold back samples to ensure there is actually an effect. You'll also want to differentiate between statistically significant results and practically significant.

- Mark as New
- Bookmark
- Subscribe
- Mute
- RSS Feed
- Permalink
- Report Inappropriate Content

Hi, Ksharp, and Reeza, I also want to understand this bit. Quite good knowledge.

Can I ask why confidence interval is a good option when the sample size is large? Confidence interval only tells if it contains the point of zero. If the sample size is large, then the confidence interval should be very narrow. Any more information from confidence interval that i don't know?

- Mark as New
- Bookmark
- Subscribe
- Mute
- RSS Feed
- Permalink
- Report Inappropriate Content

**Available on demand!**

Missed SAS Innovate Las Vegas? Watch all the action for free! View the keynotes, general sessions and 22 breakouts on demand.

What is ANOVA?

ANOVA, or Analysis Of Variance, is used to compare the averages or means of two or more populations to better understand how they differ. Watch this tutorial for more.

Find more tutorials on the SAS Users YouTube channel.