BookmarkSubscribeRSS Feed
deleted_user
Not applicable
Hello,

I am looking at attrition differences across market segments (attrition is a binary variable, 0 they left, 1 they didn't). All my my results are highly significant - even when I look at differences between two market segments that I know, from past analysis, have minimal differences in attrition. I am wondering if my huge sample size (8,000,000) is overwhelming the test. When I limit the data to a sample of 10,000, the results are more in line with what I have seen in the past, but I only want to limit the sample size if it is more accurate, not because it is giving me the results I want.


My code:

PROC NPAR1WAY WILCOXON DATA = WORK.ATTR_TEMP_SS_T2 ;
CLASS segment1;
VAR att_60day;
RUN;
2 REPLIES 2
ChuckBV
Calcite | Level 5
I'm not a statistician, just a user of statistics, but:
1. Are the proportions very different with the smaller and larger samples?
2. Is this an inference issue rather than a statistics issue per se? I recently was reading Nakagawa and Cuthill (below); they make the point (and I've seen other versions of this illustration) that, if one could measure the lengths of wings of all the birds in a population, the difference in mean wing length between sexes would almost certainly be statistically significant. Whether this would mean something to biologists is another question.


Nakagawa, S., and I. C. Cuthill. 2007. Effect size, confidence interval and statistical significance: A practical guide for biologists. Biological Reviews 82: 591-605.
Doc_Duke
Rhodochrosite | Level 12
What you are describing is the difference between "statistical significance" and "practical significance". The computation of a p-value is based on a formula that includes a function of the sample size in the denominator and a function of the "effect size" (median difference, here) in the numerator. Therefore, for any given effect size, you get a smaller and smaller p-value just by increasing the sample size.

When the sample size is really large (like yours), you have to pretty much ignore the p-value as an "index of importance" and just look at the effect size itself.

BTW, the Wilcoxon is a rather poor test if the dependent variable is binary, it loses sensitivity when there are lots of ties. You are probably better with a chi-squared test of the difference in attrition proportion across market segments.

sas-innovate-2024.png

Join us for SAS Innovate April 16-19 at the Aria in Las Vegas. Bring the team and save big with our group pricing for a limited time only.

Pre-conference courses and tutorials are filling up fast and are always a sellout. Register today to reserve your seat.

 

Register now!

What is ANOVA?

ANOVA, or Analysis Of Variance, is used to compare the averages or means of two or more populations to better understand how they differ. Watch this tutorial for more.

Find more tutorials on the SAS Users YouTube channel.

Discussion stats
  • 2 replies
  • 1117 views
  • 0 likes
  • 3 in conversation