Obtaining P-value from bootstrap samples

[Banke]

Hello everyone, 

I ran proc logistic on a bootstrapped sample, does anyone please know how I can obtain p-values? I will appreciate your help, thanks 

proc surveyselect data=melanoma.nomiss NOPRINT seed=12345
out=Bootout
method=urs /*resample with replacement*/
samprate=1 /*each bootstrap sample has N observations*/
reps=1000; /*generate 1000 bootstrap resamples */

%macro ODSOff(); /* Call prior to BY-group processing */
ods graphics off;
ods exclude all;
ods noresults;
%mend;
 
%macro ODSOn(); /* Call after BY-group processing */
ods graphics on;
ods exclude none;
ods results;
%mend;

%ODSOff;

*B. run log reg on bootstrap samples;

proc logistic data=Bootout plots=ROC outest = estimate; /** N = 1372**/
BY REPLICATE;
title "multivariable logistic regression-bootstrapped sample";
class sexcat (param=ref ref = "2") racethc(param=ref ref = "0") insurcatb (param=ref ref = "2") 
              location (param=ref ref = "0") povertycat (param=ref ref = "0") cci (param=ref ref = "0") 
              surgcat(param=ref ref = "0") hormcat (param=ref ref = "1")  radicat (param=ref ref = "0") 
              chemocat (param=ref ref = "0") histology (ref = "amelanotic") primary (ref = "head_neck");;
              
	   
model immunocat (event='1') = agenum sexcat racethc insurcatb povertycat intervalb 
                              cci location surgcat hormcat radicat chemocat histology primary
							  dxdateyearnum/ rl selection = backward include=6  
                              clodds=pl;
							  ods output CLoddsPL=CL_boot_Mort_mod_1  ;
run;

proc univariate data=cl_boot_mort_mod_1 noprint;
   class Effect;
   var  OddsRatioEst;
   output out=WidePctls1 pctlpre=P_ pctlpts=2.5 97.5 mean=Mean Std=Std P ; 
run; 


proc print data=WidePctls1 noobs label;
   format Mean Std P_2_5 P_97_5 6.4;
   label Mean="BootMean" Std="BootStdErr" P_2_5="95% Lower CL" P_97_5="95% Upper CL";
run;
%ODSOn;
proc print data = WidePctls1;
run;Output

[Rick_SAS · Accepted solution]

For a discussion of bootstrap p-values, see "How to compute p-values for a bootstrap distribution."

To compute a p-value, you need to have a hypotheses test. What is the test here? OddRatio=1?

Here are the steps:

1. Compute the statistic on the original data. Call the value T0.

2. For a one-sided p-value, compute the proportion of bootstrap statistics that exceed T0. For example,  if T0=2, you can write a DATA step to count how many bootstrap samples have OddsRatioEst > 2. The estimated p-value is (Count / NumBootSamples), where NumBootSamples=1000 for your example.

3. For a two-sided p-value, you need to figure out what value to use for the lower bound. For an odds ratio, I think the correct value is T1 = 1/T0. Then you would count how many bootstrap samples have statistics that are outside of the interval [T1, T2]. Again, the p-value is computed from this count as the proportion (Count / NumBootSamples).

[Banke]

Thank you, I will try this. Yes, my hypothesis is OR = 1

[FreelanceReinh]

Hi @Rick_SAS, I think you meant to link to your 2011 blog post "How to compute p-values for a bootstrap distribution."

Always impressive how many questions can be answered by referring to an article from your great, ever-growing blog.