Huge Sample Size and Wilcoxon 2 Sampled Test

[deleted_user]

Hello,

I am looking at attrition differences across market segments (attrition is a binary variable, 0 they left, 1 they didn't). All my my results are highly significant - even when I look at differences between two market segments that I know, from past analysis, have minimal differences in attrition. I am wondering if my huge sample size (8,000,000) is overwhelming the test. When I limit the data to a sample of 10,000, the results are more in line with what I have seen in the past, but I only want to limit the sample size if it is more accurate, not because it is giving me the results I want.


My code:

PROC NPAR1WAY WILCOXON DATA = WORK.ATTR_TEMP_SS_T2 ;
CLASS segment1;
VAR att_60day;
RUN;

[ChuckBV]

I'm not a statistician, just a user of statistics, but:
1. Are the proportions very different with the smaller and larger samples?
2. Is this an inference issue rather than a statistics issue per se? I recently was reading Nakagawa and Cuthill (below); they make the point (and I've seen other versions of this illustration) that, if one could measure the lengths of wings of all the birds in a population, the difference in mean wing length between sexes would almost certainly be statistically significant. Whether this would mean something to biologists is another question.


Nakagawa, S., and I. C. Cuthill. 2007. Effect size, confidence interval and statistical significance: A practical guide for biologists. Biological Reviews 82: 591-605.

[Doc_Duke]

What you are describing is the difference between "statistical significance" and "practical significance". The computation of a p-value is based on a formula that includes a function of the sample size in the denominator and a function of the "effect size" (median difference, here) in the numerator. Therefore, for any given effect size, you get a smaller and smaller p-value just by increasing the sample size.

When the sample size is really large (like yours), you have to pretty much ignore the p-value as an "index of importance" and just look at the effect size itself.

BTW, the Wilcoxon is a rather poor test if the dependent variable is binary, it loses sensitivity when there are lots of ties. You are probably better with a chi-squared test of the difference in attrition proportion across market segments.