topic adjusted p-value in Statistical Procedures

adjusted p-value

knighsson — Tue, 13 Apr 2021 13:44:59 GMT

Hello,

I am using proc surveyreg to compare the mean of different groups. But I do not quite sure what the adjusted p-value exactly mean in the output.

the code:

the output:

I wonder whether the adjusted p-value is due to Bonferroni adjustment? or Tukey-test?

Thank you!

Re: adjusted p-value

sbxkoenk — Tue, 13 Apr 2021 14:22:09 GMT

Hello,

You ask for a multiple-comparison adjustment of LS-means differences by the ADJUST= option on the LSMEANS statement, so you get it. 🙄

As method, you specify TUKEY (ADJUST=TUKEY) so it's a Tukey adjustment for the p-value.

More info in the doc.

https://go.documentation.sas.com/?docsetId=helpcenterwlcm&docsetTarget=home.htm&docsetVersion=1.0&locale=en

Cheers,

Koen

Re: adjusted p-value

knighsson — Tue, 13 Apr 2021 14:59:15 GMT

Thank you, But I want to know if I still want to use bonferroni adjustment, should I consider adjust p-value <0.05/6 as the significant differences?

Re: adjusted p-value

sbxkoenk — Tue, 13 Apr 2021 16:01:57 GMT

Hello,

There's no need to do a BON (Bonferroni) correction on top of the TUKEY correction.

To me a TUKEY correction is often a good choice (less conservative than Bonferroni).

But feel free to select the best correction method for your situation:

Here's the doc of the ADJUST= option (for multiple comparison p-value adjustment).

https://go.documentation.sas.com/?cdcId=pgmsascdc&cdcVersion=9.4_3.5&docsetId=statug&docsetTarget=statug_introcom_sect041.htm&locale=en#statug.introcom.lsmadjust

LSMEANS statement (across procedures):

https://go.documentation.sas.com/?cdcId=pgmsascdc&cdcVersion=9.4_3.5&docsetId=statug&docsetTarget=statug_introcom_sect040.htm&locale=en

If you do not know what to choose: choose ADJUST=BON.

Cheers,

Koen

Re: adjusted p-value

sbxkoenk — Tue, 13 Apr 2021 16:08:03 GMT

You can also read this and have a look at the recorded webinar:
What Are Best Practices for Using SAS® Survey Procedures? Q&A, Slides, and On-Demand Recording
https://communities.sas.com/t5/Ask-the-Expert/What-Are-Best-Practices-for-Using-SAS-Survey-Procedures-Q-amp-A/ta-p/706830

Re: adjusted p-value

knighsson — Tue, 13 Apr 2021 17:58:43 GMT

Thank you so much!

Re: adjusted p-value

knighsson — Tue, 13 Apr 2021 18:00:52 GMT

This is helpful! Thank you so much!

Re: adjusted p-value

knighsson — Thu, 15 Apr 2021 17:33:00 GMT

Hello, just one more question, in the table above, if the adjusted p-value <0.05, then that comparison is significantly different? Or do I need to calculate a significant level for p-value, if I used tukey test?

Thank you!

Re: adjusted p-value

StatDave — Thu, 15 Apr 2021 18:02:41 GMT

For any adjustment method, the adjusted p-values can be compared directly to the level you choose to use - such as 0.05 if that is your choice.

Re: adjusted p-value

knighsson — Thu, 15 Apr 2021 18:39:34 GMT

Thank you so much!

Re: adjusted p-value

sbxkoenk — Fri, 16 Apr 2021 08:39:11 GMT

Hello @knighsson ,

A small add-on to the comment made by @StatDave .

The adjustment method you choose does not influence in any way the significance level that you set for the hypothesis test (a comparison is also a hypothesis test). The alpha-level (for your hypothesis test) is the probability of making a type I error and is subjectively chosen by the user UPFRONT.

If you choose 0.05 as alpha, it stays that way after adjustment (with BON or TUKEY or ...) because the p-values are inflated. SAS should inflate the p-values instead of reducing the alpha because SAS does not know what the significance level / alpha is (because it's in the head of the user).

Good luck,

Koen