Dear all,
I have a dataset scored using several (external) models for predicting probabilities that a certain event occurs. That is, in the dataset I have the binary target variable as well as predicted probabilities from the models, say p_M1, p_M2, .... Now in order to compare these models, I want SAS to compute several fit statistics, like AUC, MSE, AIC.
The code below (inspired by https://blogs.sas.com/content/iml/2018/11/14/compare-roc-curves-sas.html) shows me the ROC curves, and AUC values. How do I obtain the other statistics?
proc logistic data=dataset; model target (event='1') = p_M1 p_M2 / nofit; roc 'Model 1' pred=p_M1; roc 'Model 1' pred=p_M1; run;
not necceceralily obtained by logistic regression.
I want to use
As mentioned in this note (as a method for earlier SAS releases but still usable) you can use the predicted probabilities as the sole predictor in the model to also get the ROC analysis. This method will provide the AIC statistic. If "MSE" denotes "mean square error," then this statistic doesn't apply to logistic regression since logistic models are not fit by a least squares method and therefore does not create sums of squares or mean squares.
No sure how to edit my question, of course I meant
proc logistic data=dataset; model target (event='1') = p_M1 p_M2 / nofit; roc 'Model 1' pred=p_M1; roc 'Model 2' pred=p_M2; run;
As mentioned in this note (as a method for earlier SAS releases but still usable) you can use the predicted probabilities as the sole predictor in the model to also get the ROC analysis. This method will provide the AIC statistic. If "MSE" denotes "mean square error," then this statistic doesn't apply to logistic regression since logistic models are not fit by a least squares method and therefore does not create sums of squares or mean squares.
See the link to the list of Frequently Asked-for Statistics (FASTats) in the Important Links section of the Statistical Procedures Community page. As noted there, the Brier score is available from the FITSTAT option in the SCORE statement of PROC LOGISTIC.
Are you ready for the spotlight? We're accepting content ideas for SAS Innovate 2025 to be held May 6-9 in Orlando, FL. The call is open until September 25. Read more here about why you should contribute and what is in it for you!
ANOVA, or Analysis Of Variance, is used to compare the averages or means of two or more populations to better understand how they differ. Watch this tutorial for more.
Find more tutorials on the SAS Users YouTube channel.