topic Re: Fit statistics for external models in Statistical Procedures

Fit statistics for external models

drdee — Mon, 18 Oct 2021 09:44:43 GMT

Dear all,

I have a dataset scored using several (external) models for predicting probabilities that a certain event occurs. That is, in the dataset I have the binary target variable as well as predicted probabilities from the models, say p_M1, p_M2, .... Now in order to compare these models, I want SAS to compute several fit statistics, like AUC, MSE, AIC.

The code below (inspired by https://blogs.sas.com/content/iml/2018/11/14/compare-roc-curves-sas.html) shows me the ROC curves, and AUC values. How do I obtain the other statistics?

proc logistic data=dataset;
   model target (event='1') = p_M1 p_M2 / nofit;
   roc 'Model 1' pred=p_M1;
   roc 'Model 1' pred=p_M1;
run;

not necceceralily obtained by logistic regression.

I want to use

Re: Fit statistics for external models

drdee — Mon, 18 Oct 2021 09:52:22 GMT

No sure how to edit my question, of course I meant

proc logistic data=dataset;
   model target (event='1') = p_M1 p_M2 / nofit;
   roc 'Model 1' pred=p_M1;
   roc 'Model 2' pred=p_M2;
run;

Re: Fit statistics for external models

StatDave — Mon, 18 Oct 2021 15:27:01 GMT

As mentioned in this note (as a method for earlier SAS releases but still usable) you can use the predicted probabilities as the sole predictor in the model to also get the ROC analysis. This method will provide the AIC statistic. If "MSE" denotes "mean square error," then this statistic doesn't apply to logistic regression since logistic models are not fit by a least squares method and therefore does not create sums of squares or mean squares.

Re: Fit statistics for external models

drdee — Fri, 22 Oct 2021 15:29:09 GMT

Thanks for you reply! This is useful.
I believe MSE is a useful statistic for logistic regression models as well, even though they are fit by maximum likelihood. MSE is also called the Brier score.

Re: Fit statistics for external models

StatDave — Fri, 22 Oct 2021 15:36:38 GMT

See the link to the list of Frequently Asked-for Statistics (FASTats) in the Important Links section of the Statistical Procedures Community page. As noted there, the Brier score is available from the FITSTAT option in the SCORE statement of PROC LOGISTIC.