October was a BIG month for new Python functionality in Python-sasctl and SAS Model Manager. SAS Model Manager is SAS’s leading solution for helping organizations manage, test, audit, deploy, monitor, and search their models. And yes, this includes Python models. Data scientists using Python lean on the Python-sasctl package to interact with SAS Model Manager and ensure that their models are included within a wider ModelOps and MLOps strategy. And we’ve release three new capabilities in the v1.10.0 release of python-sasctl, including a new function to assess model bias, Key Performance Indicator (KPI) and Hyperparameter integration, and support for automatic score code and metadata generation for TensorFlow Keras models. Let’s dive into each.
Assess Bias for Python Models
Python-sasctl now provides a function to assess the bias and fairness of Python models! This function leverages SAS’s Assess Bias action to take pre-scored data and determine if there are differences in the model’s accuracy and predictions across values of the specified sensitive variable, such as race or gender. The new function within python-sasctl can return the following results:
- Group metrics, which contain the model assessment statistics and average prediction for each level of the sensitive variable.
- Max differences, which contain the maximum absolute difference between all pairs of levels from the sensitive variable.
- Bias metrics, which contain the bias metrics for the predictive model.
The results of these tests can help support that a model is treating groups similarly or let the data scientist know that they need to take action to build a fair model.
The new assess model bias function can be called using the following parameters:
pzmm.JSONFiles.assess_model_bias(
score_table, # table scored by the model
actual_values, # variable with the actual target valables
sensitive_values=, # variable to assess for bias, can add more as a list
prob_values, # variables for predicted probabilities (classification)s
levels, # same order as prob_values
json_path, # directory for json files, optional
return_dataframes=True # returns group metrics, bias metrics, and max differences
)
To learn more, see this example and the following demo:
Key Performance Indicator (KPI) and Hyperparameter Integration
SAS Model Manager generates Key Performance Indicators (KPIs) automatically based on performance monitoring definitions. However, we can extend the KPIs tracked by SAS Model Manager through the addition of custom KPIs. From Python, we can perform local tests on models we’ve registered to SAS Model Manager, then pass up the resulting values as custom KPI values. For certain python models, sasctl will also generate a json file containing the hyperparameters of the model, making them easily accessible for future use.
To learn more, see this example and the following demo:
TensorFlow Keras
Keras is the high-level API of the TensorFlow platform. It provides an approachable, highly productive interface for solving machine learning (ML) problems, with a focus on modern deep learning. And now, python-sasctl can generate the metadata files and score code to run models developed using TensorFlow Keras in SAS Model Manager! To learn more, see this example and the following demo:
What would you like to see next for python-sasctl? Let us know in the comments!
To learn more about Python-sasctl and SAS Model Manager, check out the following resources:
