02-28-2015 07:12 PM
I'm a SAS novice, so thanks in advance for bearing with me...
I have straightforward 2 sample t-tests for my variables using PROC TTEST for my continuous variables and PROC FREQ for my categorical variables. I get the 95% CI for the mean and SD in the output, but I want to report confidence intervals for my p-value. Likewise, I have multivariable logistic and linear regressions for some of these variables, but can't generate 95% confidence intervals for the p-values.
A sample of my code is attached.
02-28-2015 07:26 PM
I can honestly say I've never heard of a confidence interval for a p-value.
Can you point to any statistical references for such values?
03-02-2015 12:46 PM
Hmm--I'm going to disagree a bit with that assertion, The Edwards and Berry adjustment (ADJUST=SIMULATE) in the linear modeling procs allows you to specify EPS=, where epsilon is a confidence interval on the quantile. This would establish an accuracy confidence of 100*(1 - epsilon)%.
So, the original poster could redo the analysis in PROC GLM, by changing the code to:
lsmeans illicits/cl diff=all adjust=simulate(EPS=0.05 report seed=1);
That should give some idea of a 95% CI. Note that the default value for EPS is 0.01, which would give 99% confidence.
I hope I have not marched someone over the cliff with this, though.
03-03-2015 02:02 PM
Steve, you are right that one is utilizing an aspect of the uncertainty in p with this procedure. Somehow, I doubt that this is what the OP was trying to do. But we haven't heard back....
03-03-2015 04:06 PM
The p value is random variable, dependent on the data, model being used, and so on. Under the null hypothesis, p has a uniform distribution with support on the real line from 0 to 1. But the latter fact doesn't help when the null hypothesis is false, or when you don't know whether or not H0 is true (the usual situation). Under Ha, p can have a complex distribution.