What's New with SAS Model Manager? Updated performance report

1 Like

Models can begin to degrade the moment that you put them in production. With SAS Model Manager, you can monitor the performance of your analytical models to see whether they continue to behave as expected after changes occur in market conditions or customer behavior, new data becomes available, or there is concept drift. By monitoring their performance, you can avoid model decay and re-evaluate the business value of your models in production to keep them performing at the highest levels. Put simply...if your organization focuses only on building models, and not assessing model performance over time...your organization will greatly minimize the value your Data Science and IT teams.

Updated SAS Model Manager performance monitoring report

The SAS Model Manager 2021.1.2 includes significant updates to it's out-of-the-box performance monitoring. Now you can compare performance for multiple models within the same report. A new Summary page highlights critical information about the models analyzed within the completed performance job. Quickly see if your champion model is still out-performing challenger models or other models within your project. Users can easily compare how all the different models performed over time. Simply click the different model tabs to compare performance charts like variable distribution, characteristic, stability, lift, ROC, KS and Gini per model. The performance charts now include natural language to help users better understand how to interpret their model's performance. Customization using SAS Visual Analytics embedded web components is also supported (see blog for more details). This is a great way to augment input and output variable tracking to look for changing conditions, concept drift and other issues affecting your model's performance.

Check out the differences between the original and updated SAS Model Manager performance monitoring report.

(view in My Videos)

The new Summary page contains assessment information about the models that are included in the project’s performance results. It also includes scoring observations, performance definition high-level details, and the performance rank of models within the containing project. The performance rank is the model with the best latest performance relative to the project's defined model assessment criteria. If the model assessment criteria is not specified in the project’s properties, the misclassification KPI is used to determine the rank of models in a project with a classification model function. Likewise, the average squared error (ASE) KPI is used to determine the rank of models in a project with a prediction model. Please see this blog to learn more about standard and custom KPIs supported with SAS Model Manager.

New Summary page

SAS Model Manager enables you to monitor and evaluate model performance of your champion, challenger, and other models to ensure that your models are performing efficiently. Once you create a performance definition, with as many models within a project you're interested in seeing, run it. The updated performance monitoring report will allow you to explore performance results, such as Variable Distribution, Characteristic, Stability, Lift, Gini, ROC, Kolmogorov-Smirnov (KS), Average Squared Error (ASE), and Feature Contribution (FCI) charts for each model. Now you can easily compare performance results inline for each model with natural language helping you read the results.

New individual model pages

The SAS Model Manager team has several other features are in development to further augment the updated performance monitoring report. New features include a new Compare page to visualize performance result trends for multiple models. New performance monitoring charts for multinomial models. Plus performance monitoring charts for registered/deployed SAS Visual Text Analytics models including text categories, text concepts, text sentiment and text topics. These VTA text performance monitoring trend charts are being created by our SAS Summer intern @JoeCabral. Check out what Joe created in just a few weeks.

Design ideas for text category performance monitoring charts:

Remember, SAS Model Manager is not limited to supporting just SAS models. It supports open source models too. Check out this blog about Python model support using pzmm. Here's another blog about recent enhancements to support R models.

Also check out other examples in our SAS Model Manager documentation.

Discover new SAS Viya 2021.1.2 features:

(view in My Videos)

Discover new SAS Model Manager features:

What's New with SAS Model Manager Quiz

Aside from performance monitoring report updates, what new feature is coming soon to SAS Model Manager that needs it’s own new ‘view’?

(Hint: Look closely at the SAS Model Manager images in this blog.)