topic Re: Confidence Intervals for a p-value in Statistical Procedures

Confidence Intervals for a p-value

cmhlavacek — Sun, 01 Mar 2015 00:12:12 GMT

I'm a SAS novice, so thanks in advance for bearing with me...

I have straightforward 2 sample t-tests for my variables using PROC TTEST for my continuous variables and PROC FREQ for my categorical variables. I get the 95% CI for the mean and SD in the output, but I want to report confidence intervals for my p-value. Likewise, I have multivariable logistic and linear regressions for some of these variables, but can't generate 95% confidence intervals for the p-values.

A sample of my code is attached.

Thanks!

Re: Confidence Intervals for a p-value

Reeza — Sun, 01 Mar 2015 00:26:47 GMT

I can honestly say I've never heard of a confidence interval for a p-value.

Can you point to any statistical references for such values?

Re: Confidence Intervals for a p-value

lvm — Sun, 01 Mar 2015 22:54:51 GMT

One does not calculate confidence intervals for p values.

Re: Confidence Intervals for a p-value

SteveDenham — Mon, 02 Mar 2015 17:46:27 GMT

Hmm--I'm going to disagree a bit with that assertion, The Edwards and Berry adjustment (ADJUST=SIMULATE) in the linear modeling procs allows you to specify EPS=, where epsilon is a confidence interval on the quantile. This would establish an accuracy confidence of 100*(1 - epsilon)%.

So, the original poster could redo the analysis in PROC GLM, by changing the code to:

lsmeans illicits/cl diff=all adjust=simulate(EPS=0.05 report seed=1);

That should give some idea of a 95% CI. Note that the default value for EPS is 0.01, which would give 99% confidence.

I hope I have not marched someone over the cliff with this, though.

Steve Denham

Re: Confidence Intervals for a p-value

lvm — Tue, 03 Mar 2015 19:02:16 GMT

Steve, you are right that one is utilizing an aspect of the uncertainty in p with this procedure. Somehow, I doubt that this is what the OP was trying to do. But we haven't heard back....

Re: Confidence Intervals for a p-value

PGStats — Tue, 03 Mar 2015 20:57:57 GMT

I'm not sure my poor human brain is made to handle uncertainty about uncertainty measures...

Re: Confidence Intervals for a p-value

lvm — Tue, 03 Mar 2015 21:06:42 GMT

The p value is random variable, dependent on the data, model being used, and so on. Under the null hypothesis, p has a uniform distribution with support on the real line from 0 to 1. But the latter fact doesn't help when the null hypothesis is false, or when you don't know whether or not H0 is true (the usual situation). Under Ha, p can have a complex distribution.

Re: Confidence Intervals for a p-value

SteveDenham — Wed, 04 Mar 2015 19:42:32 GMT

That would explain why the resampled values obtained with the REPORT option are all over the damn place, and not symmetric...

Steve Denham