Adjusting for multiple comparisons in Logistic Regression models

[deleted_user]

Hello, I need to adjust my p-value for significant interaction term in the adjusted logistic regression model with quite a few covariates which i have to keep in the model. I have no clue how to do that. I tried PROC Multtest but then it is not logistic and does not allow to use for covariates, and then I have no possibility to input raw p-values so my raw and bonferroni appear the same! (?). Is there any option in PROC LOGISTIC that would allow me to adjust for comparisons?
What should I do if not?
Thanks very much for any input!

[Doc_Duke]

Rather than use an adjusted p-value, you could easily use an adjusted alpha level using the Bonferroni method. If you want to protect for an overall alpha of 0.05, then just divide 0.05 by the number of tests.

An alternate that may be preferable is to use a "chunk test" to evaluate a number of interactions in one test (which would cut the number of tests in the Bonferroni method by a lot. See Frank Harrell's book; it's published by Springer.

[deleted_user]

Thank you for the input. Bonferroni method is too conservative. I am testing multiple polymorphisms (hundreds), one per model though so that each model has one polymorphism, and one- two way, and one 3 way interaction term.
Is there a simple SAS code in LOGISTCIC regression to do permutation test or FDR but utilizing the same LOGISTIC MODEL? Thanks a lot!

[StatDave · Accepted solution]

You can adjust any set of p-values by providing them to PROC MULTTEST in a data set. For example, these statements save the ParameterEstimates table to a data set in which the p-values are in a variable named ProbChiSq. PROC MULTTEST is then used to adjust them using the step-down Bonferroni method (which is less conservative):

ods output parameterestimates=pe;
proc logistic data=remiss;
model remiss(event='1') = smear blast cell infil li temp;
run;
proc multtest inpvalues(probchisq)=pe holm; run;

[deleted_user]

Wow, this actually worked! thanks so much! can i obtain fdr-adjusted p-values this way? also, can I do permutations this way? Thanks again!!!

[deleted_user]

I also have a question, if I can write macro for all those SNPs I am testing and for their multiple comparisons adjustment? Thanks!

[StatDave]

There are many p-value adjustment methods available, including FDR. See the MULTTEST documentation:

http://support.sas.com/documentation/cdl/en/statug/63033/HTML/default/multtest_toc.htm

However, resampling-based adjustment (permutation and bootstrap) cannot be done when inputting a set of p-values. These can only be done for the tests that MULTTEST does (via its TEST statement) since it must have the actual data in order to do resampling.

[deleted_user]

Thanks. Sorry for so many questions, this is the last: I realized that all the HOLM, FDR adjusted for is n covariates in the model. How would i perform adjustment, for let's say 200 other models like this that I am running yet with only one difference: different polymorphism (predictor) is included in each of those 100 models?

[StatDave]

You can adjust any set of p-values to control the experiment-wise error rate. The collection of p-values to be adjusted can be from different analyses (regression parameter tests, t-tests, etc.). A method like Holm's is always applicable as long as the p-values themselves are valid. So, you can use MULTTEST if you are doing several regressions and want to control the experiment-wise error rate across all the parameter tests.

[AnalytX]

Hi everybody,

In this case, how can we generate adjusted odd ratios and confidence intervals?

Thanks a lot for your help.

Best,