How to Create Enterprise Wide Model Dashboards with SAS Viya   Introduction   A comprehensive model dashboard should hold all information about analytical model assets of an enterprise in one place. This enables the business to react immediately to model performance degradation and save money avoids unintended harm enhances transparancy and accountabiltiy Other important aspects of the model dashboard are... automated alerting when a pre-determined threshold is hit by a model interactivity with the dashboard to dive deeper into specific details sharing the dashboard with persons outside the enterprise such as regulators But which type of information should be collected and shared within the dashboard?   Model Summary Information   The Model Summary serves as a place to communicate important details about a model to different stakeholders like model risk managers, data scientists or regulators. The goal is to create transparancy and accountability for the model development and production process by sharing information like: The model name The creator's name The creation date The target distribution The model performance The model selection process The most important variables The explainability of the model The fairness of the model regarding different groups   Model Monitoring Information   Analytical model monitoring is a crucial aspect of data science and machine learning. It involves tracking the performance of models over time to measure if they continue to make accurate predictions. This process is essential because models can become less effective as the data they were trained on becomes outdated or the environment changes. Thus, continuous model monitoring ensures that the models are continuing to provide value to the organization. In fraud, for example, criminals are very adept at quickly shifting behaviors to evade detection. Similarly in marketing, trends and buying patterns shift. Changes in business priorities, operations, customer behaviors, and data all result in the need to continually evaluate and optimize your analytical models.   Several metrics can be used to measure changes in the statistical properties of the model over time: Concept Drift: changes in the statistical properties of the target variable, which the model is trying to predict Data Drift: the distribution of the model input data changes Feature Contribution Drift: the relative contribution of model features to the prediction changes Fairness Drift: changes in model properties can affect the fairness of the model Model Decay: decrease of the accuracy of model predctions   How to Build a Comprehensive Model Dashboard on SAS Viya   The requirements of an enterprise wide model dashboard can differ from company to company. With the openess of SAS Viya you have the flexibility to build your own model dashboard based on information that is created automatically by SAS Model Studio and SAS Model Manager.   SAS Model Manager   Following we will describe the steps to collect the information for the model dashboard. The dashboard is created in SAS Visual Analytics, an interactive drag and drop interface to visualize data.   Extract and combine all model information from the SAS Model Manager repository. Assess model fairness over time Combine all model performance data Create the model dashboard report with SAS Visual Analytics (based on the data provided through step 1-3). Please note that this is just an subset of options that are available on the SAS Viya platform 1. Extract and Combine all Model Information from the SAS Model Manager Repository   A wealth of information is stored for each model registered in SAS Model Manager. In order to automatically retrieve this information for our dashboard, we need to get a list of all available models and their unique identifiers.   * Get the Viya host URL; %let viyaHost=%sysfunc(getoption(SERVICESBASEURL)); filename mm_model "&_SASWORKINGDIR./mmmodelslist.json"; * Get all models in SAS Model Manager repository; proc http method='GET' url="&viyaHost./modelRepository/models" ct='application/json' oauth_bearer = sas_services out=mm_model; run; libname mmm_lst json "&_SASWORKINGDIR./mmmodelslist.json" map='mmmodelist.map' automap=create; * Save all models in a CAS table; data public.models(promote=yes); set mmm_lst.items; run;   Besides the score code, train code and the model itself a lot of metadata is available in the repository, which is a significant differentiator from a simple Git repository. All model information is stored in JSON-files. See below an example of the files that are stored when a model is registered from SAS Model Studio.   JSON-File name Description AstoreMetadata.json In case of an SAS ASTORE model, this file keeps the information of the model location dmcas_fitstat.json Standard KPIs per specific to the model function, like AUC or ASE, seperated by partition dmcas_lift.json Lift chart values separated by partition dmcas_misc.json Misclassification values separated by partition dmcas_miscTable.json Same information as above in a different format dmcas_modelInfo.json Modeltype information dmcas_properties.json Hyperparameter settings of the algorithm dmcas_relativeimportance.json Relative variable importance for each input variable dmcas_roc.json All values to create a ROC chart separated by partition dmcas_scoreinputs.json Input variables that are expected by the model dmcas_scoreoutputs.json Output variables that are generated by the model groupMetrics.json Model performance KPIs for different levels of a sensitive variable maxDifferences.json Maximum differences between levels partialDependence[1-10] Partial dependency values for the most important variables - up to 10 different files   Note: These files may vary for different model types, model sources and option settings like assess for bias and model interpretability. You can also easily add your own model information and make it part of the model dashboard.   With the unique model identifiers we can now retrieve the required json files for each model.   %let modelID = <Replace with target model id>; filename mm_cont "&_SASWORKINGDIR./modelContent.json"; proc http url="&viyaHost./modelRepository/models/&modelID./contents?limit=50" method='GET' oauth_bearer=sas_services ct='application/json' out=mm_cont; run; libname mm_cont json "&_SASWORKINGDIR./modelContent.json" map="mm_cont.map" automap=create;   This call returns all information needed to download the files from the repository. Here is an example of the SAS code to download the JSON files:   * New File; %let modelPropertiesFileURI = <Replace with target file URI of the model properties file> filename myfile filesrvc "/files/files/&modelPropertiesFileURI."; filename myjson "&_SASWORKINGDIR./ModelProperties.json"; data _null_; infile myfile; input; file myjson; put _infile_; run;   And here is an example to create a SAS dataset from a JSON-file:   libname data3 json "&_SASWORKINGDIR./ModelProperties.json" map="&_SASWORKINGDIR./ModelProperties.map" automap=create; data public.ModelProperties(promote=yes tag="&modelID."); set data3.root; length model_id $40.; length model_function $20.; length model_algorithm $30.; length model_name $100.; length project_name $200.; length projectversion_name $20.; model_id="&modelID."; model_function="&model_function."; model_algorithm="&model_algorithm."; model_name="&model_name."; project_name="&project_name."; projectversion_name="&projectversion_name."; run;   2. Assess Model Fairness Over Time   With SAS Model Studio data scientists can assess model fairness at model development time. But what about model fairness when a model is used in production when data drift can impact the model's fairness? With SAS you can as well assess model fairness over time using the action set fairAITools and the action assessBias. Here is an example from the documentation:   proc cas; fairAITools.assessBias / cutoff='0.33', event='C', modelTable='FOREST_ASTORE', modelTableType='ASTORE', predictedVariables={'P_nominalTargetA','P_nominalTargetB','P_nominalTargetC'}, response='nominalTarget', responseLevels={'A', 'B', 'C'}, sensitiveVariable='mySensitiveVariable', table='SIMDATA'; run; quit;   When publishing to a CAS publishing destination from SAS Model Manager all models are stored in one single model table. The default name of this table is SAS_MODEL_TABLE. This will represent our production environment. Each model in the table can be accessed by it's name in an automated process.   proc cas; modelPublishing.runModelLocal / inTable={caslib="&incaslib.", name=inTableName}, modelName="&modelName.", modelTable={caslib="&mcaslib.", name="&modelTable."}, outtable={caslib="&outcaslib.", name=outTableName}; run; quit;   The assessBias action allows you to score your data and assess the model in one step. Because all available models are stored in a single model table we we need to score the data in advance by using this code:   proc cas; fairAITools.assessBias result=biasR / %if &event. ne %then %do; cutoff="&cutoff.", event="&event.", %end; modelTableType='NONE', %if &event. ne %then %do; predictedVariables={"P_&response.&nonEvent.", "P_&response.&event."}, %end; %else %do; predictedVariables={"P_&response."}, %end; response="&response.", %if &event. ne %then %do; responseLevels={"&nonEvent.", "&event."}, %end; sensitiveVariable="&sensitiveVar.", table={caslib="&outcaslib.", name=outTableName}; /* Store individual model results */ saveresult biasR['BiasMetrics'] replace caslib='CASUSER' CASOUT=bmTable; saveresult biasR['MaxDifferences'] replace caslib='CASUSER' CASOUT=mdTable; saveresult biasR['GroupMetrics'] replace caslib='CASUSER' CASOUT=gmTable; run; quit;   Note: This action is language agnostic and can score SAS, Python, and R models as long as the prescored data is avaialble. Both steps above run in a single CAS procedure code. To ensure that we just assess our model for bias we changed modelTableType to "NONE". Embedding the action calls in a macro allows us to automate the bias assessment for each model. Each iteration stores all of the action results into CAS tables for each model, each sensitive variable and each point in time. 3. Combine all Model Performance Data   With SAS Model Manager you have two options to generate model performance data for each project and different points in time. You can either use the visual interface which guides you through the performance report definition or you can use the provided SAS Model Manager macros to configure the performance reports. Both options will generate a set of CAS data sets with model performance information for each project, model and point in time. The name of the default caslib is ModelPerformanceData. Here is the list of CAS data sets that are created for a project:   Name Description MM_DATASRC_HISTORY Tracks the data set history MM_FEATURE_CONTRI_INDEX Feature contribution chart statistics for feature drift detection MM_FITSTAT Fit statistics for model decay MM_JOB_HISTORY Tracks the number and status of model performance report jobs MM_KS Kolmogorov-Smirnov statistics MM_LIFT Lift chart statistics MM_META Variable metadata roles and levels MM_ROC Receiver Operating Characteristic chart statistics MM_STD_KPI All standard KPIs MM_VAR Descriptive statistics for each variable MM_VAR_DEVIATION Detailed deviation index statistics for data drift detection MM_VAR_SUMMARY Summarized deviation index statistics for data drift detection     As mentioned before all CAS data sets are stored in one single caslib for each project. How to avoid that the datasets are not overwritten if the name stays the same? Did you know that you can tag datasets? This is done by SAS Model Manager automatically. Each CAS data set gets a two level name. The first level is the tag with the unique project UUID and the second level is the name of the performance data set.   Model Performance Data Sets   To work with tagged CAS data sets you need to create a CAS libref with the tag statement.   Tagged Libref   We would like to create a dashboard for all model, so we need to combine/append all of the individual model performance datasets into single datasets for each type. The tag information will be stored in an added variable to distinguish between all models in the overall model dashboard report that we will create with SAS Visual Analytics. 4. Create the model dashboard report with SAS Visual Analytics   Interactive model dashboards provide the insights and details necessary for the organization to identify issues with analytical models quickly and take corrective action. Often, organizations have many models running in their production systems and need a high-level model health overview dashboard. If issues arise the dashboard needs to enable the users to drill down into the performance details of a model to identify the root cause.   Summary SAS Model Manager and SAS Visual Analytics are powerful tools that can help you create and manage enterprise-wide model information, which are concise summaries of the performance, fairness, and explainability of your machine learning models. In this blog, we outlined how to use the data from SAS Model Manager and the visualization capabilities of SAS Visual Analytics to build a model dashboard that meets the needs and expectations of your organization. SAS Viya offers a flexible framework that allows you to customize the design, graphics, and insights of your dashboard according to your specific requirements. If you are interested in further details, we recommend this article about Model Reporting. ... View more

Model Studio is a user friendly, low-code visual frontend that allows you to build, automate and operationalize Machine Learning pipelines in a streamlined way, saving time and effort. As a supplement SAS provides a REST API, called Automated Machine Learning in short MLPA, which allows users to automate and industrialize Machine Learning pipelines in batch-mode. MLPA can be used in two scenarios: 1. Create Machine Learning pipelines automatically, including smart data preparation steps. 2. Run a pre-defined pipeline template on different data segments and different target variables. In the following blog, I will demonstrate how to automate and industrialize Machine Learning pipelines by using pre-defined pipelines and custom steps in SAS Studio. Within the custom steps I generate code for several REST calls to    retrieve a list of available pipeline templates, create automation projects for different data segments, get the status of the automation projects, and get the results from an automation project.   But wait! What is a pipeline template? You can build model pipelines in Model Studio and store them to the "Exchange" as a template. For more information see here.   Examples   1. Retrieve Pipeline Templates   filename resp "%substr(&mypath.,11)/resp.json"; proc http method="GET" url="https://servername/mlPipelineAutomation/pipelineTemplates?filter=and(eq(modifiedBy,'viyademo01'))" ct="application/vnd.sas.collection+json" oauth_bearer = sas_services out=resp; run; libname resp json "%substr(&mypath.,11)/resp.json" map="resp.map" automap=create;   2. Create Automation Project   Create JSON payload file: proc json out=filepl pretty; write open object; write values "dataTableUri" "/dataTables/dataSources/cas~fs~cas-shared-default~fs~Public/tables/&tablename."; write values "type" "predictive"; write values "name" "&targetname."; write values "description" "&targetname. automatic project"; write values "settings"; write open object; write values "autoRun" &auto.; write values "modelingMode" "Standard"; write values "applyGlobalMetadata" &global.; write values "maxModelingTime" 100; write close; write values "pipelineBuildMethod" "template"; write values "analyticsProjectAttributes"; write open object; write values "targetVariable" "&targetname."; write values "partitionEnabled" true; write values "classSelectionStatistic" "&statistic."; write close; write values "links"; write open array; write open object; write values "method" "GET"; write values "rel" "initialPipelineTemplate"; write values "href" "/mlPipelineAutomation/pipelineTemplates/&templateid."; write values "uri" "/mlPipelineAutomation/pipelineTemplates/&templateid."; write values "type" "application/octet-stream"; write close; write close; write close; run;   Create automation project with payload above:   proc http method="POST" url="https://servername/mlPipelineAutomation/projects" ct="application/vnd.sas.analytics.ml.pipeline.automation.project+json" in=filepl oauth_bearer = sas_services out=model; run;   3. Get Project Status   proc http method="GET" url="https://servername/mlPipelineAutomation/projects?filter=eq(modifiedBy,'viyademo01')" ct="application/vnd.sas.collection+json" oauth_bearer = sas_services out=autop; run;   4. Get Results   filename reports "%substr(&mypath.,11)/reports.json"; proc http method="GET" url="https://servername/mlPipelineAutomation/projects/f7cacaf3-092b-45d3-9c7a-d72d2ed764b8/models/@championModel/reports" ct="application/vnd.sas.collection+json" oauth_bearer = sas_services out=reports; run; libname reports json "%substr(&mypath.,11)/reports.json" map="reports.map" automap=create;   The following video shows how the custom steps are used in a SAS Studio flow to create one pipeline per segment of a data set:     SAS is continuing to enhance Model Studio to be more flexible in areas like multiple target specification, but until those features are released there are also ways you can help yourself here using the MLPA API and custom steps in SAS Studio. So, this is just an example of what can be done with MLPA.   More information can be found here: Automated Machine Learning Python example ... View more

An Application for Explainable AI in Forecasting   Table of Content   Part I: Machine Learning in Forecasting  Part II: Explainability Methods in Forecasting Part III: An Application for Explainable AI in Forecasting  Introduction Example Report Client 1. Calling the casManagement API 2. Receiving Data from SAS Visual Analytics (VA) 3. Executing the SAS Job Server Conclusion    Introduction   In this third part of our blog series we will show you how to create an easy to use application to explain Forecasting models on the fly. We will use SAS Visual Analytics, the Data Driven Content object and the SAS Job Execution Engine to create this application. Some things need to be done on the client side and some on the server side. We will explain each step on the client side and on the server side. Example Report Before we go deeper into the technical detail of each of the different steps, let’s have a look at how a typical report that uses our application would look like.     As we see from the video this interactive report would be a tool that would be used by reporting analysts, planners, management and other business personas to get a deep insight into the drivers that have an impact on the final forecasts.   Note: The example report shows forecasts where historical data is also available but it would work exactly the same with no limitations for future forecasts as well since the method is focusing on explaining the prediction of a machine learning model without taking actual data into consideration.   Now let’s have a look at how you can build a similar report and tailor it to the business needs of your organisation.    Client   Let's first take a look at SAS Visual Analytics. Within SAS Visual Analytics there is a huge selection of different visualization objects available to create reports, insights or analytical models. The Data Driven Content object let s you add customized visualizations and run custom code. In the screenshot below you can see where to select this object on the left hand side of SAS Visual Analytics. In the middle you can see a HTML form that let you select the settings to run the code on the server side. The content of the Data Driven Conent object is included by the URL property of this object shown on the right hand side of the screenshot.   The HTML torm enables the user to select and enter/autofill all the settings that are needed to run the job code. The settings are selected through pre-populated dropdowns and entered/autofilled in an input field. In total the following information is needed: The CAS-Library (1 & 3) in which the Input table and the Model table are located, the Input table (2) and the Model table (4) that are being used, the ID column (5) of the Input table and the value of the ID (6).   The implementation of this application has three major parts: For the pre-populated dropdowns (1-5) we call the casManagement APIs To preserve automated actions on objects (autofill the query ID) we need to receive the data of SAS Visual Analytics Executing the SAS Job   1. Calling the casManagement API   The API calls are done relative to the winows.origin property of the server so that the code should run dynamically in any Viya installation. The calls are made utilizing the fetch function: fetch( window.origin + '/casManagement/providers/cas/sources/cas-shared-default/children?limit=100&sortBy=name%3Aascending&start=0', { // mode: 'no-cors', method: 'GET', headers: { Accept: 'application/json', }, } ).then((response) => { if (response.ok) { // Do stuff here }))   As the pre-loaded dropdowns depend on the users' selection the calls are chained and when one of the parent value changes the child data is refreshed as well. 2. Receiving Data from SAS Visual Analytics (VA)   In our application we would like to give the user the option to select a specific observation through a line chart and autofill the "ID to Explain" field (6). In order to achieve automated actions between standard visualization objects and the data driven content object, the object needs to receive data from VA. The SAS VA Messaging Uti allows you to receive the data. The implementation is simple as the ID field is unique.   function onDataReceived(resultData) { ID.value = resultData.data[0]; } va.messagingUtil.setOnDataReceivedCallback(onDataReceived); 3. Executing the SAS Job   Once the user clicks on the Calculate SHAP values button (7) a spinner overlay is added (to give a visual indication that the job is running) and the SAS Job gets called with all the values the user provided in the form:   fetch(jeeURI, { method: 'GET', }) .then((resp) => { console.log('Response'); SPINNER.style.display = 'none'; FORM.style.display = 'block'; }) .then((body) => { console.log('Success'); }) .catch((error) => { console.log('Error'); }); });   On a response the spinner is overlay is hidden again and the user can interact with the form again. This allows the user, after an initial set up of the main parameters (1-5), to quickly iterate through more IDs. Note: The code above is written in Javascript and included inline in the HTML form.   Server As already described in the section above the following values are captured by the client:   Variable name in the code Description caslib_ds CASLIB of the scoring table name_ds Scoring table caslib_m CASLIB of the model table (score code) name_m Model table (score code) id_name Unique ID variable id Query ID   The first step loads the model into memory, to make sure that it is available for execution. You can load data into memory with proc casutil.  After the model is loaded, the model's input and output information is extracted from the model to reduce user inputs: proc astore; ods output InputVariables=work.inputs; ods output OutputVariables=work.outputs; describe rstore=&caslib_m..&name_m.; run;   The query table will be create based on the id variable and it's value: data casuser.query; set &caslib_ds..&name_ds.(where=(&id_name=&id)); run; After all necessary information is available, a CASL program is generated and stored in a .sas-program based on the information collected above. The program needs to be generated on the fly because every model can have different predictor and outcome variables. In addition, a model can have interval and nominal predictors. In this case the syntax of the code will differ slightly.   Here are two examples of the generated code. One for the global explanation of Forecasting models and one for the local explanation of Forecasting models. The theory of these methods is explained in the second part of our blog series.   Global Explanation Example Code   In this example the preset parameter is set to 'GlobalReg' and the dataGeneration method is set to 'None', which would be the default in this case. Details can be found here. explainModel.linearExplainer result=globr / table = {name='PRICEDATA_ID', caslib='PUBLIC'} query = {name='QUERY', caslib='CASUSER'} modelTable = {name='GB_PRICEDATA_MODEL_ID', caslib='MODELS'} modelTableType = 'ASTORE' predictedTarget = 'P_sale' seed=1234 preset = 'GlobalReg' dataGeneration = {method='None'} inputs= {{name = "sale_lag3"}, {name = "sale_lag2"}, {name = "sale_lag1"}, {name = "discount"}, {name = "price"}} ;run; send_response({gr=globr['ParameterEstimates']});run;   The generated program will be referenced by the filename pgm_g and  executed by the runCASL action: proc cas; session casauto; loadactionset "explainModel"; loadactionset "sccasl"; source pgm_g; %include pgm_g; endsource; run; sccasl.runCASL result=r / code=pgm_g; run; describe r; param_est=findtable(r); saveresult param_est dataout=work.parameterEstimates; quit;   Local Explanation Example Code   We slightly change the parameters to perform a local explanation: explainModel.linearExplainer result=shapr / table = {name='PRICEDATA_ID', caslib='PUBLIC'} query = {name='QUERY', caslib='CASUSER'} modelTable = {name='GB_PRICEDATA_MODEL_ID', caslib='MODELS'} modelTableType = 'ASTORE' predictedTarget = 'P_sale' seed =1234 preset = 'KERNELSHAP' dataGeneration = {method='None'} inputs= {{name = "sale_lag3"}, {name = "sale_lag2"}, {name = "sale_lag1"}, {name = "discount"}, {name = "price"}} ;run; send_response({sr=shapr['ParameterEstimates']});run; The result tables will be stored/refreshed to CAS and can be accessed and visualized by the SAS Visual Analytics report everytime  a user clicks the "Execute" button.   Conclusion This concludes our series of articles. We hope you enjoyed reading them and found them useful. We've touched upon many different topics so don't despair if some areas are still unclear - this is perfectly normal. What we hope you take out from these articles is a good understanding of Machine Learning techniques in forecasting and how you can cleverly apply existing state-of-the-art techniques to explain your forecasts. In the last article, we've also shown how you can create a custom application to dynamically run explainability techniques on the fly using SAS Viya's powerful capabilities. The final application provides strong insights in your forecasts and is extremely easy to use by any business stakeholder. As a last thought we want to mention that explainability in forecasting has attracted the attention of many researchers recently and we expect a lot of new exciting developments in this area. As always, we'll make sure to keep you updated!       ... View more

Explainability Methods in Forecasting   Table of Content       Part I: Machine Learning in Forecasting Part II: Explainability Methods in Forecasting  Introduction What are Shapley Values? Using the linearExplainer Action for Time Series Data Part III: An application for Explainable AI in Forecasting     Introduction   In this second part of our blog series we focus on SHAP[1] as local explanation method and how to apply this method to the ABT table we created from the time series data. As we mentioned in part I the challenge we are facing is that this method works well when the variables we are using are independent from each other, so we have to find a way to adapt our methodology to overcome this issue.   But before we start, let's give you a short introduction into Shapley Values.   What are Shapley Values?   The concept of Shapley Values came from economists[2] for game theory. They tried to solve the problem of award distribution among multiple team members.   How to fairly attribute member's contribution? The solution by Lloyd Shapley satisfies the following properties: EFFICIENCY: All individual awards should add up to the total earning DUMMY: If including an individual brings no additional earning in any situation, then this individual should receive zero award SYMMETRY: If including two individuals add the same amount of additional earnings, then they should receive the same award ADDITIVITY: If including individual A inceases the earning by the same amount of two other individuals B and C, then A should receive the sum of B's and C's award. The Shapley Value is the ONLY solution that satisfies all constraints! It is based on a weighted marginal contribution of a member among all possible coalitions. But wait what is a coalition and what is a marginal contribution?     What would be the weighted marginal contribution among all possible coalitions? Here is an example Shapley Value for member A:     Written as formula:     See: https://christophm.github.io/interpretable-ml-book/shapley.html    In the formula above, p is the total number of members and S is the number of members in the coalition excluding the member of interest. The weight is inversely proportional to the size of a coalition “group” where each “group” includes all coalitions with the same number of members. So, in our example above we have 4 groups: Group 1: Adding 0 other person, size 1  Group 2: Adding 1 other person, size 3  Group 3: Adding 2 people, size 3   Group 4: Adding 3 people, size 1  Each group ends up in having the same total weight of 1/4 and all weights add up to 1.   This approach can be transferred to explain the prediction for a (local) observation. Each feature value of the observation is a member in a game where the prediction is the award. The calculation of Shapley Values is computationally expensive as it requires the evaluation of the model with all possible coalitions/combinations of features. There are faster approximation methods available, like SHAP[1].   This SHAP method is implemented in the SAS action linearExplainer which is one action of the Explain Model action set. A good explanation of the SHAP method can be found in the book Interpretable Machine Learning. In the next section we will explain how to adapt the linearExplainer action for time series data. Using the linearExplainer Action for Time Series Data   The standard KERNELSHAP preset implementation of the SAS action is following these steps: Pick a single observation (query) Generate random observations by sampling from each variable's distribution separately Apply the model score code that was generated by a previous step to the new observations Weight the observations based on their coalitions Run a weighted linear regression on model's prediction Interpret the linear regression model coefficients   Because of the way we built our analytical base table (ABT) - transferring it from transactional to one row per subject - our features are not independent from each other. For details, please refer to part I of our blog series. When we would apply step 2 - generating random observations - the dependency among the features would get lost. To preserve the dependency structure, it is possible to suppress the random sampling process 😊. Here is an example code: proc cas; explainModel.linearExplainer result=shapr / table = {name='PRICEDATA_ID', caslib='PUBLIC'} query = {name='QUERY', caslib='CASUSER'} modelTable = {name='GB_PRICEDATA_MODEL_ID', caslib='MODELS'} modelTableType = 'ASTORE' predictedTarget = 'P_sale' seed =1234 preset = 'KERNELSHAP' dataGeneration = {method='None'} inputs= {{name = "sale_lag3"}, {name = "sale_lag2"}, {name = "sale_lag1"}, {name = "discount"}, {name = "price"}} ; run; So, by adding the line of code "dataGeneration = {method='None'}", random sampling will be suppressed and the model's score code will be applied to the original observations. This preserves the feature dependencies and let you explain the prediction of machine learning models like Gradient Boosting or other tree based algorithms like LightGBM in our Forecasting case. However, please note that    the accuracy depends on how well the original data cover the coalitions, the Shapley values of highly correlated features may bleed into each other, this method can be seen as approximation of the Shapley coalition/cohort values in [3]. Note: If you are interested in a global explanation of your machine learning model for time series data, you can just adapt the preset parameter to 'GLOBALREG' to create a surrogate model for a global explanation of your model.   In our third and last part of this blog series, we will show how to explain forecasting models globally and locally in an application.   See you in part III! References [1] Lundberg, S. M., and Lee, S.-I. (2017). “A Unified Approach to Interpreting Model Predictions.” In Advances in Neural Information Processing Systems 30 (NIPS 2017), 4765–4774. La Jolla, CA: Neural Information Processing Systems Foundation.   [2] Shapley, Lloyd S. “A value for n-person games.” Contributions to the Theory of Games 2.28 (1953): 307-317   [3] Mase et al. 2019 "Explaining black box decisions by Shapley cohort refinement"           ... View more

Interested in ModelOps with SAS Model Manager and not sure where to get started? Please take a look at this is short video explaining how to register a Python model into SAS Model Manager and deploying the Python model for in-memory batch scoring using SAS.   SAS Model Manager provides support for the analytical model lifecycle for SAS and Open Source models.  The software provides capabilities for common model storage, metadata, search, selection, testing, deployment, monitoring, KPI-based alerting, and dashboards. These functions enable easy model access and integration by data scientists, business users, and information technology (IT) teams.   There is more to a model than the pickle file you created. To run your model in production, you need to know: what input variables are needed by your model A good best practice is providing acceptable range of values for each input variable, based on your testing data. what output variables the model generates A good best practice is providing what "good looks like" and the expected distribution by output variables, based on your testing data. For instance, expected probability of 1s is 90% and 0s is 10%.   SAS Model Manager's Model Repository API registers all model assets. To facilitate use of the API, SAS created the Python Zip Model Management (PZZM) package, available on the sassoftware organization on GitHub.   From this central, governed hub for all model sources and types, the model is published to a CAS destination. CAS is one of multiple deployment options. Other destinations include SAS' real-time scoring Micro Analytics Service (MAS) and containers. More details are available in the documentation.   The video below demonstrates how to apply the model to new data using theSWAT package. The SWAT package allows Python users to make a connection to CAS and run CAS actions.     The three steps outlined here lay the foundation of a solid, maintainable and enterprise ready ModelOps environment. Please note, creating an enterprise ready system for ModelOps requires more details. Hans-Joachim Edert and I discussed several aspects during of the process in the following meetup video:   ... View more