What's New with SAS Model Manager? Generate standard and custom KPIs

2 Likes

Want to use common statistical measures to track model performance over time? SAS Model Manager includes a new MMKPI macro and action set to generate out-of-the-box standard KPIs and run user-provided KPIs. Now standard and custom key performance indicators (KPIs) can be defined, computed, and used to generate alerts. Check out this article to see how KPIs are integrated in a new out-of-the-box workflow-based email notification when model performance exceeds the project's model assessment criteria threshold value.

A KPI is a computed value derived from the SAS Model Manager performance job or from a user's own data and jobs. Common KPIs for classification projects and interval prediction projects are "Misclassification" and "Average squared error" respectively. The number of possible custom KPI metrics is limitless. Here are the current standard KPIs generated by the the new SAS Model Manager MMKPI macro and action set:

Average Squared Error (_ASE_)
Area Under the ROC Curve (_AUC_)
Cumulative Lift at a Depth of 5 (_Cumlift5_)
Cumulative Lift at a Depth of 10 (_Cumlift10_)
Cumulative Lift at a Depth of 15 (_Cumlift15_)
Cumulative Lift at a Depth of 20 (_Cumlift20_)
Cumulative Lift at a Depth of 30 (_Cumlift30_)
F1 Score (_F1_)
False Positive Rate (_FPR_)
Misclassification (_MISC_)
R-Squared (_RSquared_)
True Positive Rate (_TPR_)

Example output of the new MMKPI macro and action set:

With the 2021.1.1 (May 2021) stable release of SAS Viya, SAS Model Manager includes a new Files tab to run the standard KPIs for a project, once a performance monitoring report job has run successfully. Notice the project UUID (_projectID) is automatically populated for you. Simply click Run for the ProjectKPI.sas file and View results to see the output (shown above) code and log for generating standard out-of-the-box KPIs.

Additional enhancements to the new Files tab came with the 2021.1.3 (July 2021) stable release with support for custom KPIs. Need some help writing custom KPIs? Please see this link for some examples.

Generate standard out-of-the-box KPIs and custom KPIs directly within SAS Model Manager:

Results of these standard KPIs are already integrated into an out-of-the-box, simplified model monitoring alert notification workflow and the updated performance monitoring results. Soon, you'll see KPIs used in model health dashboards, model and project comparisons, alerts, and notifications.

New SAS Model Manager performance monitoring report including latest model assessment criteria KPIs:

See the SAS Model Manager documentation for more information about the new MMKPI macro and action set. Essentially, the %MM_KPI_ACTIONSET macro enables users to:

create standard KPI data table from the performance monitoring results using the runKPI action set
add custom KPI data and labels to the standard KPI table using the addCustomKPI action set, and/or
run user-provided custom KPI code using the runCustomKPI action set

In addition to running standard KPIs, the upcoming stable release of SAS Model Manager will allow users to integrate and run custom KPIs directly within the new Files tab. In the interim, leverage the new MMKPI action set via APIs and via SAS code within SAS Studio.

In SAS Studio, here's the code to start a new CAS session and activate the new MM KPI action set.

/* Start a CAS session and enable new MM KPI action set */
cas _mmcas_ uuidmac=session_uuid;
%put session _mmcas_ uuid: &session_uuid;

caslib _ALL_ assign;

/* update string with your Project UUID */
%let projectUUID=%nrstr(23a309cc-7451-47b3-a667-7976ed9d726e);
libname mmkpi cas caslib="ModelPerformanceData" tag="&projectUUID.";

%mm_kpi_actionSet;

Here's the code to generate standard out-of-the-box MM KPIs using this new action set:

/* Generate Model Manager Standard KPI Table (<projectUUID.>MM_STP_KPI)
   Need to have already run the performance job first */

proc cas;
_projectID = "&projectUUID.";
builtins.loadactionset / actionSet="mmkpi";

mmkpi.runKPI result=r status=s /
   projectUUID = _projectID
   casout={
        caslib="ModelPerformanceData",
        name=_projectID || ".MM_STD_KPI",
        promote=True}
;

if dim(r.Error) > 0 then do;
   describe r;
   print r.Error;
end;

run;
quit;

Here's a very simple example of creating custom KPIs using the new KPI action set. You have lots of flexibility for creating custom KPIs.

/* Generate custom KPIs to calculate the rate of change from one time period to next for 
       Misclassification (MISC), 
       Average Square Error (ASE), and 
       Area under the curve (AUC)
   and append the custom KPIs with the standard MM KPI table */

data work.MM_USER_KPI;
   set MMKPI.MM_STD_KPI;
run;

proc sort data=work.MM_USER_KPI;
by ModelUUID TimeSK;
run;

data work.MM_USER_KPI;
   set work.MM_USER_KPI;
   by ModelUUID;

   /* put your custom code here */
   MISC_RateChange = (_MISC_ - lag(_MISC_))/lag(_MISC_);
   ASE_RateChange = (_ASE_ - lag(_ASE_))/lag(_ASE_);
   AUC_RateChange = (_AUC_ - lag(_AUC_))/lag(_AUC_);

run;

/* Load your new custom KPIs into an in-memory table. This code ensures MM variables are loaded with varchar format, not character. */

data casuser.MM_USER_KPI (promote=Yes);
set work.MM_USER_KPI (rename=(ModelName=ModelNameOLD ModelUUID=ModelUUIDOLD ProjectUUID=ProjectUUIDOLD TimeLabel=TimeLabelOLD));
length ModelName varchar(18);
length ModelUUID varchar(36);
length ProjectUUID varchar(36);
length TimeLabel varchar(2);
ModelName=ModelNameOLD;
ModelUUID=ModelUUIDOLD;
ProjectUUID=ProjectUUIDOLD;
TimeLabel=TimeLabelOLD;
drop ModelNameOLD ModelUUIDOLD ProjectUUIDOLD TimeLabelOLD;
run;

/* Integrate the custom KPIs back into the MM project KPI table */

proc cas;
_projectID = "&projectUUID.";
mmkpi.addCustomKPI result=r /
	kpiTable = {caslib='casuser', name='MM_USER_KPI'}
	projectUUID = _projectID
	;
run;
quit;

Additional custom KPI sample code is available on the Open Model Manager samples page on GitHub.

Also check out other examples in our SAS Model Manager documentation.

Discover new SAS Viya 2021.1.1 features

(view in My Videos)