How to Create Enterprise Wide Model Dashboards with SAS Viya

7 Likes

How to Create Enterprise Wide Model Dashboards with SAS Viya

Introduction

A comprehensive model dashboard should hold all information about analytical model assets of an enterprise in one place. This

enables the business to react immediately to model performance degradation and save money
avoids unintended harm
enhances transparancy and accountabiltiy

Other important aspects of the model dashboard are...

automated alerting when a pre-determined threshold is hit by a model
interactivity with the dashboard to dive deeper into specific details
sharing the dashboard with persons outside the enterprise such as regulators

But which type of information should be collected and shared within the dashboard?

Model Summary Information

The Model Summary serves as a place to communicate important details about a model to different stakeholders like model risk managers, data scientists or regulators. The goal is to create transparancy and accountability for the model development and production process by sharing information like:

The model name
The creator's name
The creation date
The target distribution
The model performance
The model selection process
The most important variables
The explainability of the model
The fairness of the model regarding different groups

Model Monitoring Information

Analytical model monitoring is a crucial aspect of data science and machine learning. It involves tracking the performance of models over time to measure if they continue to make accurate predictions. This process is essential because models can become less effective as the data they were trained on becomes outdated or the environment changes. Thus, continuous model monitoring ensures that the models are continuing to provide value to the organization. In fraud, for example, criminals are very adept at quickly shifting behaviors to evade detection. Similarly in marketing, trends and buying patterns shift. Changes in business priorities, operations, customer behaviors, and data all result in the need to continually evaluate and optimize your analytical models.

Several metrics can be used to measure changes in the statistical properties of the model over time:

Concept Drift: changes in the statistical properties of the target variable, which the model is trying to predict
Data Drift: the distribution of the model input data changes
Feature Contribution Drift: the relative contribution of model features to the prediction changes
Fairness Drift: changes in model properties can affect the fairness of the model
Model Decay: decrease of the accuracy of model predctions

How to Build a Comprehensive Model Dashboard on SAS Viya

The requirements of an enterprise wide model dashboard can differ from company to company. With the openess of SAS Viya you have the flexibility to build your own model dashboard based on information that is created automatically by SAS Model Studio and SAS Model Manager.

SAS Model Manager

Following we will describe the steps to collect the information for the model dashboard. The dashboard is created in SAS Visual Analytics, an interactive drag and drop interface to visualize data.

Extract and combine all model information from the SAS Model Manager repository.
Assess model fairness over time
Combine all model performance data
Create the model dashboard report with SAS Visual Analytics (based on the data provided through step 1-3).

Please note that this is just an subset of options that are available on the SAS Viya platform

1. Extract and Combine all Model Information from the SAS Model Manager Repository

A wealth of information is stored for each model registered in SAS Model Manager. In order to automatically retrieve this information for our dashboard, we need to get a list of all available models and their unique identifiers.

* Get the Viya host URL;
%let viyaHost=%sysfunc(getoption(SERVICESBASEURL)); 

filename mm_model "&_SASWORKINGDIR./mmmodelslist.json";

* Get all models in SAS Model Manager repository;
proc http 
	method='GET'
	url="&viyaHost./modelRepository/models"
	ct='application/json'
	oauth_bearer = sas_services
	out=mm_model; 
run;

libname mmm_lst json "&_SASWORKINGDIR./mmmodelslist.json" map='mmmodelist.map' automap=create;

* Save all models in a CAS table;
data public.models(promote=yes);
	set mmm_lst.items;
run;

Besides the score code, train code and the model itself a lot of metadata is available in the repository, which is a significant differentiator from a simple Git repository. All model information is stored in JSON-files. See below an example of the files that are stored when a model is registered from SAS Model Studio.

JSON-File name	Description
AstoreMetadata.json	In case of an SAS ASTORE model, this file keeps the information of the model location
dmcas_fitstat.json	Standard KPIs per specific to the model function, like AUC or ASE, seperated by partition
dmcas_lift.json	Lift chart values separated by partition
dmcas_misc.json	Misclassification values separated by partition
dmcas_miscTable.json	Same information as above in a different format
dmcas_modelInfo.json	Modeltype information
dmcas_properties.json	Hyperparameter settings of the algorithm
dmcas_relativeimportance.json	Relative variable importance for each input variable
dmcas_roc.json	All values to create a ROC chart separated by partition
dmcas_scoreinputs.json	Input variables that are expected by the model
dmcas_scoreoutputs.json	Output variables that are generated by the model
groupMetrics.json	Model performance KPIs for different levels of a sensitive variable
maxDifferences.json	Maximum differences between levels
partialDependence[1-10]	Partial dependency values for the most important variables - up to 10 different files

Note: These files may vary for different model types, model sources and option settings like assess for bias and model interpretability. You can also easily add your own model information and make it part of the model dashboard.

With the unique model identifiers we can now retrieve the required json files for each model.

%let modelID = <Replace with target model id>;
filename mm_cont "&_SASWORKINGDIR./modelContent.json";

proc http
	url="&viyaHost./modelRepository/models/&modelID./contents?limit=50"
	method='GET'
	oauth_bearer=sas_services
	ct='application/json'
	out=mm_cont;
run;

libname mm_cont json "&_SASWORKINGDIR./modelContent.json" map="mm_cont.map" automap=create;

This call returns all information needed to download the files from the repository. Here is an example of the SAS code to download the JSON files:

* New File;
%let modelPropertiesFileURI = <Replace with target file URI of the model properties file>
filename myfile filesrvc "/files/files/&modelPropertiesFileURI.";
filename myjson  "&_SASWORKINGDIR./ModelProperties.json";
 
data _null_;
	infile myfile;
	input;
	file myjson;
	put _infile_;
run;

And here is an example to create a SAS dataset from a JSON-file:

libname data3 json "&_SASWORKINGDIR./ModelProperties.json" map="&_SASWORKINGDIR./ModelProperties.map" automap=create;
 
data public.ModelProperties(promote=yes tag="&modelID.");
	set data3.root;
	length model_id $40.;
	length model_function $20.;
	length model_algorithm $30.;
	length model_name $100.;
	length project_name $200.;
	length projectversion_name $20.;

	model_id="&modelID.";
	model_function="&model_function.";
	model_algorithm="&model_algorithm.";
	model_name="&model_name.";
	project_name="&project_name.";
	projectversion_name="&projectversion_name.";
run;

2. Assess Model Fairness Over Time

With SAS Model Studio data scientists can assess model fairness at model development time. But what about model fairness when a model is used in production when data drift can impact the model's fairness?

With SAS you can as well assess model fairness over time using the action set fairAITools and the action assessBias. Here is an example from the documentation:

proc cas;
	fairAITools.assessBias /
	    cutoff='0.33',
	    event='C',
	    modelTable='FOREST_ASTORE',
	    modelTableType='ASTORE',
	    predictedVariables={'P_nominalTargetA','P_nominalTargetB','P_nominalTargetC'},
	    response='nominalTarget',
	    responseLevels={'A', 'B', 'C'},
	    sensitiveVariable='mySensitiveVariable',
	    table='SIMDATA';
	run;
quit;

When publishing to a CAS publishing destination from SAS Model Manager all models are stored in one single model table. The default name of this table is SAS_MODEL_TABLE. This will represent our production environment. Each model in the table can be accessed by it's name in an automated process.

proc cas;
	modelPublishing.runModelLocal / 
	     inTable={caslib="&incaslib.", name=inTableName},
	     modelName="&modelName.",
	     modelTable={caslib="&mcaslib.", name="&modelTable."},
	     outtable={caslib="&outcaslib.", name=outTableName};
	run;
quit;

The assessBias action allows you to score your data and assess the model in one step. Because all available models are stored in a single model table we we need to score the data in advance by using this code:

proc cas;
	fairAITools.assessBias result=biasR /
	    %if &event. ne  %then %do;
		    cutoff="&cutoff.",
		    event="&event.",
	    %end;
	    modelTableType='NONE',
	    %if &event. ne  %then %do;
	    	predictedVariables={"P_&response.&nonEvent.", "P_&response.&event."},
	    %end;
	    %else %do;
	    	predictedVariables={"P_&response."},
	    %end;
	    response="&response.",
	    %if &event. ne  %then %do;
	    	responseLevels={"&nonEvent.", "&event."},
	    %end;
	    sensitiveVariable="&sensitiveVar.",
	    table={caslib="&outcaslib.", name=outTableName};

	/* Store individual model results */
	saveresult biasR['BiasMetrics'] replace caslib='CASUSER' CASOUT=bmTable;
	saveresult biasR['MaxDifferences'] replace caslib='CASUSER' CASOUT=mdTable;
	saveresult biasR['GroupMetrics'] replace caslib='CASUSER' CASOUT=gmTable;
	run;
quit;

Note: This action is language agnostic and can score SAS, Python, and R models as long as the prescored data is avaialble. Both steps above run in a single CAS procedure code. To ensure that we just assess our model for bias we changed modelTableType to "NONE".

Embedding the action calls in a macro allows us to automate the bias assessment for each model. Each iteration stores all of the action results into CAS tables for

each model,
each sensitive variable and
each point in time.

3. Combine all Model Performance Data

With SAS Model Manager you have two options to generate model performance data for each project and different points in time. You can either use the visual interface which guides you through the performance report definition or you can use the provided SAS Model Manager macros to configure the performance reports.

Both options will generate a set of CAS data sets with model performance information for each project, model and point in time. The name of the default caslib is ModelPerformanceData. Here is the list of CAS data sets that are created for a project:

Name	Description
MM_DATASRC_HISTORY	Tracks the data set history
MM_FEATURE_CONTRI_INDEX	Feature contribution chart statistics for feature drift detection
MM_FITSTAT	Fit statistics for model decay
MM_JOB_HISTORY	Tracks the number and status of model performance report jobs
MM_KS	Kolmogorov-Smirnov statistics
MM_LIFT	Lift chart statistics
MM_META	Variable metadata roles and levels
MM_ROC	Receiver Operating Characteristic chart statistics
MM_STD_KPI	All standard KPIs
MM_VAR	Descriptive statistics for each variable
MM_VAR_DEVIATION	Detailed deviation index statistics for data drift detection
MM_VAR_SUMMARY	Summarized deviation index statistics for data drift detection

As mentioned before all CAS data sets are stored in one single caslib for each project. How to avoid that the datasets are not overwritten if the name stays the same? Did you know that you can tag datasets? This is done by SAS Model Manager automatically. Each CAS data set gets a two level name. The first level is the tag with the unique project UUID and the second level is the name of the performance data set.

Model Performance Data Sets

To work with tagged CAS data sets you need to create a CAS libref with the tag statement.

Tagged Libref

We would like to create a dashboard for all model, so we need to combine/append all of the individual model performance datasets into single datasets for each type. The tag information will be stored in an added variable to distinguish between all models in the overall model dashboard report that we will create with SAS Visual Analytics.

4. Create the model dashboard report with SAS Visual Analytics

Interactive model dashboards provide the insights and details necessary for the organization to identify issues with analytical models quickly and take corrective action. Often, organizations have many models running in their production systems and need a high-level model health overview dashboard. If issues arise the dashboard needs to enable the users to drill down into the performance details of a model to identify the root cause.

(view in My Videos)

Summary

SAS Model Manager and SAS Visual Analytics are powerful tools that can help you create and manage enterprise-wide model information, which are concise summaries of the performance, fairness, and explainability of your machine learning models. In this blog, we outlined how to use the data from SAS Model Manager and the visualization capabilities of SAS Visual Analytics to build a model dashboard that meets the needs and expectations of your organization. SAS Viya offers a flexible framework that allows you to customize the design, graphics, and insights of your dashboard according to your specific requirements.

If you are interested in further details, we recommend this article about Model Reporting.