BookmarkSubscribeRSS Feed
rhysticlight
Calcite | Level 5

Hello,

I am using proc GLIMMIX to develop a model for mortality by a specified time-point (binary outcome of alive or dead at 30-days). The data are clustered by hospital so I am using PROC GLIMMIX to fit a random intercept model for the data using the reporting hospital as the subject variable.

In order to properly evaluate the performance of my models I would like to examine their calibration and discrimination. I found this article (41364 - ROC analysis for binary response models fit in the GLIMMIX, NLMIXED, GAM or other procedures) detailing how to create an ROC curve and get a c-statistic (i.e. area under the ROC curve) for examining model discrimination; however, I am still having problems figuring out how to get a good measure of calibration.

Proc GLIMMIX does not have the LACKFIT option to produce a Hosmer-Lemeshow statistic as in Proc Logistic (and I am fairly certain that this statistic is not appropriate to use with clustered data anyway). I am trying to figure out how to do something along the lines of producing a "plot of expected vs. observed mortality rates across deciles of increasing risk" but am having some trouble figuring out how to go about doing this.

Any help you could provide woul dbe greatly appreciated.

Thank you!

1 REPLY 1
SteveDenham
Jade | Level 19

The hard part is interpreting anything like an HL stat in light of the clustering.  I would suggest doing a within-hospital lack of fit test for each hospital, as well as one overall that essentially ignored the clustering.  If the latter shows a lack of fit, it might then be quickly identified as being due to a specific hospital.  I think all of these tests would have to be obtained from PROC LOGISTIC, first with a by hospital statement, and then without.

You might be able to take the within hospital p values as data for a generalized linear model with a beta distribution, and use the sample sizes as weights.  This might provide a better pooled representation than the "ignore the clustering" approach.

Good luck.

Steve Denham

sas-innovate-2024.png

Join us for SAS Innovate April 16-19 at the Aria in Las Vegas. Bring the team and save big with our group pricing for a limited time only.

Pre-conference courses and tutorials are filling up fast and are always a sellout. Register today to reserve your seat.

 

Register now!

What is ANOVA?

ANOVA, or Analysis Of Variance, is used to compare the averages or means of two or more populations to better understand how they differ. Watch this tutorial for more.

Find more tutorials on the SAS Users YouTube channel.

Discussion stats
  • 1 reply
  • 3057 views
  • 0 likes
  • 2 in conversation