BookmarkSubscribeRSS Feed

Assess Bias of Python Models, Automatically Create Scoring Code for TensorFlow Keras Models and More

Started a month ago by
Modified a month ago by
Views 320

October was a BIG month for new Python functionality in Python-sasctl and SAS Model Manager. SAS Model Manager is SAS’s leading solution for helping organizations manage, test, audit, deploy, monitor,  and search their models. And yes, this includes Python models. Data scientists using Python lean on the Python-sasctl package to interact with SAS Model Manager and ensure that their models are included within a wider ModelOps and MLOps strategy. And we’ve release three new capabilities in the v1.10.0 release of python-sasctl, including a new function to assess model bias, Key Performance Indicator (KPI) and Hyperparameter integration, and support for automatic score code and metadata generation for TensorFlow Keras models. Let’s dive into each.


Assess Bias for Python Models


Python-sasctl now provides a function to assess the bias and fairness of Python models! This function leverages SAS’s Assess Bias action to take pre-scored data and determine if there are differences in the model’s accuracy and predictions across values of the specified sensitive variable, such as race or gender.  The new function within python-sasctl can return the following results:

  • Group metrics, which contain the model assessment statistics and average prediction for each level of the sensitive variable.
  • Max differences, which contain the maximum absolute difference between all pairs of levels from the sensitive variable.
  • Bias metrics, which contain the bias metrics for the predictive model.

The results of these tests can help support that a model is treating groups similarly or let the data scientist know that they need to take action to build a fair model.


The new assess model bias function can be called using the following parameters:


        score_table, # table scored by the model
        actual_values, # variable with the actual target valables
        sensitive_values=, # variable to assess for bias, can add more as a list
        prob_values, # variables for predicted probabilities (classification)s
        levels, # same order as prob_values
        json_path, # directory for json files, optional
        return_dataframes=True # returns group metrics, bias metrics, and max differences


To learn more, see this example and the following demo:



Key Performance Indicator (KPI) and Hyperparameter Integration


SAS Model Manager generates Key Performance Indicators (KPIs) automatically based on performance monitoring definitions. However, we can extend the KPIs tracked by SAS Model Manager through the addition of custom KPIs.  From Python, we can perform local tests on models we’ve registered to SAS Model Manager, then pass up the resulting values as custom KPI values. For certain python models, sasctl will also generate a json file containing the hyperparameters of the model, making them easily accessible for future use.


To learn more, see this example and the following demo:



TensorFlow Keras


Keras is the high-level API of the TensorFlow platform. It provides an approachable, highly productive interface for solving machine learning (ML) problems, with a focus on modern deep learning. And now, python-sasctl can generate the metadata files and score code to run models developed using TensorFlow Keras in SAS Model Manager! To learn more, see this example and the following demo:



What would you like to see next for python-sasctl? Let us know in the comments!


To learn more about Python-sasctl and SAS Model Manager, check out the following resources:


This is so cool! Can't wait to try it!

Version history
Last update:
a month ago
Updated by:



Registration is open! SAS is returning to Vegas for an AI and analytics experience like no other! Whether you're an executive, manager, end user or SAS partner, SAS Innovate is designed for everyone on your team. Register for just $495 by 12/31/2023.

If you are interested in speaking, there is still time to submit a session idea. More details are posted on the website. 

Register now!

Free course: Data Literacy Essentials

Data Literacy is for all, even absolute beginners. Jump on board with this free e-learning  and boost your career prospects.

Get Started

Article Tags