Setting up customized KPI metrics to monitor your models over time in SAS Viya: a step-by-step guide
It has never been so easy to develop a machine learning model. However, machine learning is more than just model development. It even goes much further than model deployment, i.e., scheduling and running a model in production. Machine learning is a continuous improvement process ensuring that the model does not only run well today, but also on long term. The moment you put models in production they might start to degrade. Therefore, productionized models need to be closely monitored and action needs to be taken if performance is going down.
SAS Viya’s Model Manager offers a wide range of performance metrics to monitor machine learning models, and has the ability to design and execute a governance process on what action to take when performance does not meet the predefined requirements.
In this article, I will focus on setting up customized performance metrics for model monitoring in SAS Model Manager. Although a wide variety of standard out-of-the-box performance monitoring capabilities are available, some machine learning projects require specific metrics, often set by business-defined KPIs. As from SAS Viya 4 (2021.1.3), it is possible to define your own customized KPI metrics for model monitoring for any model under consideration, centralizing performance monitoring of all productionized models in one place.
An overview of the out-of-the-box performance metrics include charts to assess input and output data drift, lift and ROC charts and many more. A detailed list can be found here. Those metrics are automatically generated when you create a new performance definition. The process to set up standard performance charts can be found here. The calculation of the performance metrics can be easily scheduled. However, the question still remains how do you set up your customized KPI metrics in SAS Model Manager?
All the magic happens under the Files tab in SAS Model Manager (#1). Within this tab, you have the ability to specify custom code that will execute every time the performance reports are updated. Per default, a Standard KPI file is provided (#2) with SAS Model Manager and calculates the following statistics for each time stamp:
- Average Squared Error (_ASE_)
- Area Under the ROC Curve (_AUC_)
- Cumulative Lift at a Depth of 5 (_Cumlift5_)
- Cumulative Lift at a Depth of 10 (_Cumlift10_)
- Cumulative Lift at a Depth of 15 (_Cumlift15_)
- Cumulative Lift at a Depth of 20 (_Cumlift20_)
- Cumulative Lift at a Depth of 30 (_Cumlift30_)
- F1 Score (_F1_)
- False Positive Rate (_FPR_)
- Misclassification (_MISC_)
- R-Squared (_RSquared_)
- True Positive Rate (_TPR_)
Note that the Standard KPI file invokes the action set %MM_KPI_ACTIONSET (#3), containing the runKPI action (#4) which computes the aforementioned statistics.
Although the file is executed each time the performance reports are updated, the user has the option to re-run the KPI files upon request. Clicking on the RUN button next to the file (#5), will execute the code and update the KPI metrics. Important to know is that the Standard KPI file uses data generated by the performance reports. Therefore, it is needed to run the performance reports at least once before the Standard KPI file can be execute on demand. After execution, the results can be viewed under the files tab (#6). Additionally, a performance report with the standard KPIs metrics is added under the Performance tab (#7), calculated for each time stamp and each model that is being monitored. Note that the code is not only executed for SAS models, but can also be applied to open source models.
Next to the standard KPIs, customized KPI metrics can be defined. Under the Files tab, you can specify your own customized KPI calculation (#8). Using the ‘+’-sign (#9), any .sas code file can be added to the calculation. In my example, I have chosen to report the precision and recall of the top decile with the highest probability to be diagnosed Parkinson. As the execution of the KPI files happens after the execution of the performance reports, all performance data is available for the calculation of custom KPI metrics. To calculate the precision and recall of the top 10% highest scored observations, the data from the lift charts can be re-used. More specifically, the Cumulative Percent response and Cumulative Captured Response of the 10th percent is selected, representing the precision and recall at the first decile. Remark that the code to calculate the customized KPI metrics should be written in CAS language (CASL). Although I have used some pre-calculated performance statistics for the calculation of custom KPIs, any formula can be used and any data set can be exploited and included in the code.
Here is the code from the custom KPI file in SAS Model Manager:
/* Code for Custom KPI file in SAS Model Manager */
inputTable = {caslib='ModelPerformanceData', name=projectUUID || '.mm_lift'};
table.fetch result = LiftTable / to = 10000 table = {name=inputTable.name, caslib= inputTable.caslib};;
describe(LiftTable);
NewStatistic = LiftTable.Fetch.where(Percent = 10)[,{"ProjectUUID", "ModelUUID", "ModelName", "TimeSK", "CuPercentResp","CuCapturedResp","CollectionDate"}];
saveresult NewStatistic casout=outputTable.name caslib=outputTable.caslib replace;
/* modify column attribute as need */
table.alterTable status=s2 / columns={
{name="CollectionDate" rename='Datetime', label='Created Datetime'}
{name='CuPercentResp', rename='PrecisionDepth10', label='Precision at Depth 10'}
{name='CuCapturedResp', rename='RecallDepth10', label='Recall at Depth 10'}}
caslib=outputTable.caslib name=outputTable.name;
/* End – Code for Custom KPI file in SAS Model Manager */
You can test and develop the customized KPI code in SAS Studio by adding the cas statements and defining the project universally unique identifier of the SAS Model Manager project and an temporary output table.
/* Add those lines to execute code in SAS Studio */
cas;
caslib _all_ assign;
proc cas;
projectUUID="735f5aa8-b84c-4c4d-91b6-8bd85f8950ff";
outputTable={caslib='casuser', name='MM_MYSTATISTIC'};
/* End – add */
/* Code for Custom KPI file in SAS Model Manager */
inputTable = {caslib='ModelPerformanceData', name=projectUUID || '.mm_lift'};
table.fetch result = LiftTable / to = 10000 table = {name=inputTable.name, caslib= inputTable.caslib};;
describe(LiftTable);
NewStatistic = LiftTable.Fetch.where(Percent = 10)[,{"ProjectUUID", "ModelUUID", "ModelName", "TimeSK", "CuPercentResp","CuCapturedResp","CollectionDate"}];
saveresult NewStatistic casout=outputTable.name caslib=outputTable.caslib replace;
/* modify column attribute as need */
table.alterTable status=s2 / columns={
{name="CollectionDate" rename='Datetime', label='Created Datetime'}
{name='CuPercentResp', rename='PrecisionDepth10', label='Precision at Depth 10'}
{name='CuCapturedResp', rename='RecallDepth10', label='Recall at Depth 10'}}
caslib=outputTable.caslib name=outputTable.name;
/* End – Code for Custom KPI file in SAS Model Manager */
/* Add those lines to execute code in SAS Studio */
run;
quit;
proc print data=casuser.mm_mystatistic; run;
/* End – add */
Under the performance tab, you can now add a customized graph to visualize the results. Go to the graph of the Standard KPI, and select ‘Explore and Visualize’ (#10).
Next, design the desired graph in SAS Visual Analytics with the new calculated statistics and save the report. On the snowman menu of the design graph, select Copy link… (#11), enable Embeddable web component (#12) and copy the link (#13).
Going back to Performance tab in SAS Model Manager, select the snowman menu > Edit layout > Model Charts (#14). Select Output Variable Tracking (#15) and add a custom chart (#16). Finally, paste the link of the graph designed in SAS Visual Analytics in the report object link (#17).
Save the results. The Custom KPI is now added to the performance reports. Any time the performance reports re-run, the standard and custom KPI statistics are automatically updated.
