Confidence Intervals for a p-value

[cmhlavacek]

I'm a SAS novice, so thanks in advance for bearing with me...

I have straightforward 2 sample t-tests for my variables using PROC TTEST for my continuous variables and PROC FREQ for my categorical variables. I get the 95% CI for the mean and SD in the output, but I want to report confidence intervals for my p-value. Likewise, I have multivariable logistic and linear regressions for some of these variables, but can't generate 95% confidence intervals for the p-values.

A sample of my code is attached.

Thanks!

[Reeza]

I can honestly say I've never heard of a confidence interval for a p-value. 

Can you point to any statistical references for such values?

[lvm]

One does not calculate confidence intervals for p values.

[SteveDenham]

Hmm--I'm going to disagree a bit with that assertion, The Edwards and Berry adjustment (ADJUST=SIMULATE) in the linear modeling procs allows you to specify EPS=, where epsilon is a confidence interval on the quantile.  This would establish an accuracy confidence of 100*(1 - epsilon)%.

So, the original poster could redo the analysis in PROC GLM, by changing the code to:

lsmeans illicits/cl diff=all adjust=simulate(EPS=0.05 report seed=1);

That should give some idea of a 95% CI.  Note that the default value for EPS is 0.01, which would give 99% confidence.

I hope I have not marched someone over the cliff with this, though.

Steve Denham

[lvm]

Steve, you are right that one is utilizing an aspect of the uncertainty in p with this procedure. Somehow, I doubt that this is what the OP was trying to do. But we haven't heard back....

[PGStats]

I'm not sure my poor human brain is made to handle uncertainty about uncertainty measures...

PG

[lvm]

The p value is random variable, dependent on the data, model being used, and so on. Under the null hypothesis, p has a uniform distribution with support on the real line from 0 to 1. But the latter fact doesn't help when the null hypothesis is false, or when you don't know whether or not H0 is true (the usual situation). Under Ha, p can have a complex distribution.

[SteveDenham]

That would explain why the resampled values obtained with the REPORT option are all over the damn place, and not symmetric...

Steve Denham