Fit statistics for external models

[drdee]

Dear all,

I have a dataset scored using several (external) models for predicting probabilities that a certain event occurs. That is, in the dataset I have the binary target variable as well as predicted probabilities from the models, say p_M1, p_M2, .... Now in order to compare these models, I want SAS to compute several fit statistics, like AUC, MSE, AIC.

The code below (inspired by https://blogs.sas.com/content/iml/2018/11/14/compare-roc-curves-sas.html) shows me the ROC curves, and AUC values. How do I obtain the other statistics?

proc logistic data=dataset;
   model target (event='1') = p_M1 p_M2 / nofit;
   roc 'Model 1' pred=p_M1;
   roc 'Model 1' pred=p_M1;
run;

not necceceralily obtained by logistic regression.

I want to use 

[drdee]

No sure how to edit my question, of course I meant

proc logistic data=dataset;
   model target (event='1') = p_M1 p_M2 / nofit;
   roc 'Model 1' pred=p_M1;
   roc 'Model 2' pred=p_M2;
run;

[StatDave · Accepted solution]

As mentioned in this note (as a method for earlier SAS releases but still usable) you can use the predicted probabilities as the sole predictor in the model to also get the ROC analysis. This method will provide the AIC statistic. If "MSE" denotes "mean square error," then this statistic doesn't apply to logistic regression since logistic models are not fit by a least squares method and therefore does not create sums of squares or mean squares.

[drdee]

Thanks for you reply! This is useful.
I believe MSE is a useful statistic for logistic regression models as well, even though they are fit by maximum likelihood. MSE is also called the Brier score.

[StatDave]

See the link to the list of Frequently Asked-for Statistics (FASTats) in the Important Links section of the Statistical Procedures Community page. As noted there, the Brier score is available from the FITSTAT option in the SCORE statement of PROC LOGISTIC.