With the new bias and fairness metrics in the Model Card on SAS Model Manager, it was only a matter of time before bias and fairness charts were added to the performance monitoring report. As of 2025.03, the wait is over! Users can now see how a model’s performance on fairness and bias changes over time. With this additional information, users can now include measures of model bias when making their decision to refresh, replace, or retrain their production models.
Measuring Model Fairness and Bias
The new fairness and bias section in the performance monitoring report displays the following metrics across each time interval:
- Demographic parity – the maximum pairwise difference for the proportion of observations that are binned into the event level, given the cut-off value
- Equal Accuracy - maximum pairwise difference across values of the selected variable for Accuracy
- Equalized Odds- the larger of the maximum pairwise differences across values of the selected variable for True Positive Rate and False Positive Rate
- Equal Opportunity- the maximum pairwise difference across values of the selected variable for Ture Positive Rate
- Predictive Parity- maximum pairwise difference in the predicted variable corresponding to the event level
Demographic Parity, Predictive Parity, Equal Accuracy, Equalized Odds, and Equal Opportunity are calculated using SAS’s Assess Bias Action. There is also a Group Unfairness Index (GUI), calculated using the approach outlined by Szepannek and Lübke in their paper Facing the Challenges of Developing Fair Risk Scoring Models.
The specific variables shown depend on the target type of the model. Additionally, for each value of a selected variable, the following is calculated and charted:
- Observation count
- Average prediction
- Various measures of model performance
This allows for quick and easy comparison of how the model may treat each group in a protected class. For example, users may find critical insights, like:
- If the data fed to our model under-represents specific age groups
- If our model gives men a higher average prediction over women
- If our model is more accurate on white individuals over black individuals
With an understanding of the model’s current measures of bias, teams can be better informed and take appropriate action.
Adding Fairness and Bias to Performance Monitoring Reports
To add fairness and bias metrics to performance monitoring reports of classification and prediction models written in SAS, Python, or R, you must first mark variables as “Assess for Bias” in the Project’s Variable tab. The variables marked as “Assess for Bias” can be inputs to a model in the project or not used by any project models at all. The variables must be available in the data used by performance monitoring, even if the variables are not used by the model to make a prediction. While an organization may not want protected class information to influence a model’s output, that information must be available to truly validate that the model is not treating protected classes differently. Note that if a variable was marked as “Assess for Bias” in SAS Model Studio, that designation will automatically carry over to SAS Model Manager upon model registration.
Additionally, fairness and bias Key Performance Indicators (KPIs) thresholds are available in a Project’s Properties tab. These KPI thresholds are used to assess model performance in the model card for classification and prediction models.
Once variables are marked as “Assess for Bias”, create and run a new performance monitoring definition. Once completed, scroll down to see the new Fairness and Bias section.
The following demo video goes over Monitoring Model Fairness from SAS Model Studio to Performance Monitoring and the Model Card in SAS Model Manager.
Want to learn more about building Responsible AI applications? Check out the following resources:
