Model Cards were added to SAS Model Manager in July 2024 and our team is continuing to add enhancements and broaden their scope. The first iteration of our Model Cards focused on prediction and classification models developed in SAS Model Studio and models developed using Python. With the release of SAS Viya 2024.11, we are now adding support to build a complete model card for SAS classification and prediction models trained in SAS Studio through updates to our Import Model macros.   The SAS Model Manager macros provide SAS programmers the ability to interact with SAS Model Manager from SAS code. SAS programmers just need to add values for the various parameters and then run the code. The Import Model Macro and the Import ASTORE Model Macro have been used to take models developed in SAS code and register them directly into SAS Model Manager, along with their various metadata files. And with our latest release, this macro now generates all the files to start building a complete model card.   Within these macros, two new parameters were added, and one existing parameter is now required. The TRAINTABLE parameter was previously optional and specifies the model’s training data. This parameter is now required and is used to calculate relative importance for the model’s input variables. The two new parameters, SENSITIVEVAR and VARIMPTABLE, are both optional. The new SENSITIVEVAR parameter is used to specify a variable to use for assessing model fairness. For each value of the sensitive variable, SAS calculates the model’s accuracy and average prediction to determine if the model is treating classes differently. This uses SAS’s assess bias action under the hood. You can read more about the assess bias action in the documentation. Since tree-based models have a built-in mechanism for determining variable importance, the VARIMPTABLE parameter is used to specify the variable importance table from a tree-base model.   To have the most robust model card, we recommend that you split your data into training, testing, and validation sets prior to model training. When leveraging the macro, specify the training percentage using the SAMPLEPCT parameter and the testing percentage via the SAMPLEPCT2 parameter. Splitting your data into training, testing, and validation sets allows more points of comparison within the model card.   Once the models are registered into SAS Model Manager, users can add additional information through their interactions with the model and project in SAS Model Manager. You can learn more about these steps in the last article on Model Cards.    To see the updated Import Model and Import ASTORE Model macros in action, check out this short demo:   Next for the model card, we are evaluating updating PROC REGISTERMODEL so SAS programmers on SAS Viya Workbench can register a model into SAS Model Manager and generate a complete model card. We are also evaluating exporting the model card as a PDF. But what would you like to see next for the SAS Model Card? Leave a comment below! ... View more

Model cards were introduced with the SAS Viya 2024.07 release and act as a nutrition label for AI models. The model cards in SAS Viya feature easy-to-understand visuals and actionable takeaways. When designing the model card, SAS made sure it contained useful and digestible information for a variety of stakeholders and would become a natural extension of the model’s lifecycle. As you develop and manage models within SAS Viya, the model card starts to populate. But what about Python models developed in environments outside of SAS Viya? Don’t worry, we didn’t forget about our Python modelers! In Part 1 of this series, we explored creating a model card for models in SAS Model Studio, and in this article, we will focus on Python models developed outside of SAS Viya. To build a complete model card, we will use the python-sasctl package and tools on SAS Viya. This example notebook includes the code you need to generate the files for the model card.       Model Card example     Getting Everything in SAS Model Manager   SAS Viya cannot automatically track actions taken outside of the platform, but data scientists training their models in their Python environments outside of SAS Viya can use several functions in python-sasctl to generate the files required for SAS Model Manager and the model card. They can also import their model into SAS Model Manager directly from their Python notebook or code. Once they are happy with their trained model, these are the functions from python-sasctl our Python-wielding data scientists will need to run: pzmm.PickleModel.pickle_trained_model() to create a binary representation (pickle) of their model. pzmm.JSONFiles.write_var_json() using their input data to write out metadata about the variables the model expects as inputs. pzmm.JSONFiles.write_var_json() using their output data to write out metadata about the variables the model generates as outputs. pzmm.JSONFiles.write_model_properties_json() to save key information about the model, including target, algorithm, description, and modeler. pzmm.JSONFiles.write_file_metadata_json() for file assignments. pzmm.JSONFiles.calculate_model_statistics() to save model performance metrics at the time of training. pzmm.JSONFiles.assess_model_bias() to examine the model’s performance and prediction across levels of a potentially sensitive variable. This step won’t apply if there are no variables available for bias assessment, but you can learn more about his function in this notebook. pzmm.JSONFiles.generate_model_card() to generate the remaining files needed by the model card. pzmm.JSONFiles.create_requirements_json() to specify the dependencies required for the Python model. This step is optional but recommended for easy dependency management. You can learn more about this function in this notebook. pzmm.ImportModel.import_model() to write the model score code and register the files you created into SAS Model Manager. Be sure to specify a project using the project parameter! Optionally, you can write your own score code, zip all the files up using pzmm.ZipModel.zip_files and import them using model_repository.import_model_from_zip.   Now, you should see your Python model in SAS Model Manager and a nearly completed model card! We'll walk through each section below and highlight any remaining tasks to complete the card.    The Model Card   The first step for making your model card is to register your model into a project. The pzmm.ImportModel.import_model() imports models into SAS Model Manager directly from a Python notebook or code file. This function has a project parameter where a user can specify which project the model should be registered. If the project doesn’t exist, python-sasctl will automatically create a project with the given name and register the model to it. Once your models are registered, you can open your models in SAS Model Manager.   The Model Card will appear as the first tab of a model instance inside SAS Model Manager when the model function is prediction or classification. You can set the model function using the target type parameter of the pzmm.JSONFiles.generate_model_card() function, but you can also update the property in SAS Model Manager directly.   Left-Hand Pane   The left-hand pane contains three sets of information: tags, the modeler, and the responsible party.  The modeler is specified using the modeler parameter of the pzmm.JSONFiles.write_model_properties_json()  function, but you can update the modeler and add custom tags to the model in the Properties tab of the model. The responsible party is the user or group responsible for the model and this field is populated at the project level. A link to the project is available near the top of the screen. Within the project, navigate to the Properties Tab and then the Model Usage section. You can update the responsible party here as well as any model usage fields.    Updating the Model Card left-hand pane   Overview   The Overview section provides an at-a-glance review of the model. It provides visuals of model health that are easy to understand and are supported by other sections of the card. This tab reports on model performance during training and over time. It also reports on influential variables, variable privacy classifications, and completeness of model card. Most of the data in the Overview section will be populated using the files created earlier,  but there are areas that will require attention when first registered.   If you are noticing a blue warning about the thresholds for your training metrics, you can review and update the thresholds for action in the project properties under model evaluation. For the performance monitoring metrics to become available, complete the steps listed in the Model Audit section of this article. To ensure the variable privacy classifications come through, scroll down to the Data Summary and complete the steps I’ve listed in the corresponding section below. If you want the “No” to change to a “Yes” in the Limitations Documented block, then complete the Model Usage section of the card, as outlined in the next section of this article. Finally, to see fairness metrics, you needed to run pzmm.JSONFiles.assess_model_bias() .   This Overview section can help provide evidence that your model is in good health or direct your attention towards areas that need attention.   Model Usage   The Model Usage section describes the intended usage, expected benefits, out-of-scope use cases, and limitations of the model. This section must be filled out manually at either the project or model level in the Properties tab. The values of the model usage are inherited from the project-level properties. However, model-specific information can be specified for each property value at the model level.    Updating the Model Card Model Usage section   Data Summary   The Data Summary section provides information about the training data from SAS Information Catalog.  When registering your model from Python, the training data property will be set automatically if you specify the train_data parameter using the pzmm.JSONFiles.generate_model_card()  function, but you may be prompted to run an analysis in the Data Summary section. If you see a button in this section, click it to run your analysis.   Once the analysis is complete, you should see a summary courtesy of SAS Information Catalog. This summary contains the number of columns, number or rows, size, status, completeness of data, information privacy classifications, data tags, and data descriptions. If there are gaps in the description, tags, and status, then these can be corrected in SAS Information Catalog. Working with data in SAS Information Catalog may require advanced permissions, so work with your data engineers or data owners to ensure your data metadata is complete. Having complete data about your data is a best practice for building trustworthy models!    Updating the Model Card Data Summary section Model Summary   The Model Summary section examines the model’s performance at the time of training, which corresponds to the training donut charts in the Overview tab. The information in this tab is populated by running the pzmm.JSONFiles.calculate_model_statistics()  function.  The Model Summary section includes information about the model target, algorithm, development tool and version, various accuracy measures across training, testing, and validation splits, generalizability, and variable importance. If you’ve also run the pzmm.JSONFiles.assess_model_bias(), fainress metrics will also appear in this tab. Overall, this tab provides a ton of information about how well the model was performing during training which can provide a baseline for monitoring model performance over time.        Complete Model Card Summary section   Model Audit   While the Model Summary section focuses on the model’s performance at the time of training, the Model Audit section reports on performance over time.  The Model Audit section also provides a deeper dive into the performance monitoring donut charts in the Overview section.   The Model Audit section relies on two capabilities of SAS Model Manager: performance monitoring and Key Performance Indicator (KPI) rules. You will set your KPI thresholds in the project properties under model evaluation. You can create a rule in just a few clicks, as outlined in the documentation or the quick demo below:   Creating KPI alert rules   Performance monitoring reviews model performance over time in batch at user-defined time points. To create a performance monitor report, you need data with the actual or ground-truth values, and SAS Model Manager graphs model performance over time. Without the ground-truth, you will only get metrics on data drift. You can create a performance monitoring report in a few moments from the project, as outlined in these steps or the demo video below:   Creating and running a performance monitoring definition    Now, you can view the latest model accuracy, fairness, and model drift metrics against your thresholds on the latest run of performance monitoring.     Creating a complete Model Card   Pulling it All Together   Even if your Python model was trained outside of SAS Viya, you can still build a complete model card! To learn more, visit the example notebook and see an example of each function called out in the article. Now you should have everything you need to create a complete nutrition label for your Python model. Post your questions, feedback, and ideas in the comments below!   ... View more

With the release of SAS Viya 2024.07, a model card is now available in SAS Model Manager. The model card in SAS Model Manager was built to be like a nutrition label for AI models. What sets the SAS Model Card apart from previous model cards is the use of descriptive visuals, to make model cards accessible to all personas involved in the analytics process, including data scientists, data engineers, MLOPs engineers, managers, executives, risk managers, business analytics, end-users, and any other stakeholder with access to the SAS Viya environment. Additionally, most of our model card populates automatically as the model is developed, managed, and deployed in SAS Viya.   Model Card example   The first release of the model card supports classification and prediction models from various sources. As users and teams develop and manage their model within SAS Viya, more of the model card will automatically populate. The model card brings together information about the training data, model performance at the time of training, and model performance over time. So, for this article, we will review how to generate a complete model card for models from SAS’s no-code / low-code interface, SAS Model Studio. But fear not! This article is just the first of a series and we will focus on Python models in part 2.   The Model Card   The first step for making your model card is to register your model. From SAS Model Studio, you can register models from the Data Mining and Machine Learning pipelines in the Pipeline Comparison tab. Select the models you want to register, then open the options menu and click “Register models”. Once your models are registered, you can open your models in SAS Model Manager.   Registering a model in SAS Model Studio   The Model Card will appear as the first tab of a model instance inside SAS Model Manager when the model function is prediction or classification. If you’ve registered your model from SAS Model Studio, ensure that you are running SAS Viya 2024.07 (or later) and that your model has the function property listed as prediction or classification.    Model Card immediately after registration   Left-Hand Pane   The left-hand pane contains three sets of information: tags, the modeler, and the responsible party.  The modeler will populate automatically for models registered from SAS Model Studio, The responsible party is the user or group responsible for the model and this field is populated at the project level. A link to the project is available near the top of the screen. Within the project, navigate to the Properties Tab and then the Model Usage section. You can update the responsible party here as well as any model usage fields.    Updating the Model Card left-hand pane Overview   The Overview section provides an at-a-glance review of the model. It provides visuals of model health that are easy to understand and are supported by other sections of the card. This section reports on model performance during training and over time. It also reports on influential variables, variable privacy classifications, and completeness of the model card. Most of the data in the Overview section will be populated automatically for models coming from Model Studio, but there are areas that will require attention when first registered.   If you are noticing a blue warning about the thresholds for your training metrics, you can review and update the thresholds for action in the project properties under model evaluation. We encourage users to update these thresholds to ensure they are appropriate for their current use case, as thresholds for action may differ from one use case to the next. For the performance monitoring metrics to become available, complete the steps listed in the Model Audit section of this article. To ensure the variable privacy classifications come through, scroll down to the Data Summary and complete the steps listed in the corresponding section below. If you want the “No” to change to a “Yes” in the Limitations Documented block, then complete the limitations in the Model Usage section of the card, as outlined in the next section of this article. Finally, to see fairness metrics, before you ran your pipelines, you needed to select a variable to Assess for Bias in Model Studio.   This Overview section can help provide evidence that your model is in good health or direct your attention towards areas that need help.   Complete Model Card Overview section   Model Usage   The Model Usage section describes the intended usage, expected benefits, out-of-scope use cases, and limitations of the model. This section must be filled out manually at either the project or model level in the Properties tab. The values of the model usage are inherited from the project-level properties. However, model-specific information can be specified for each property value at the model level.    Updating the Model Card Model Usage section Data Summary   The Data Summary section provides information about the training data from SAS Information Catalog. When registering your model from SAS Model Studio, the training data property should already be populated, but you may be prompted to run an analysis in the Data Summary section. If you see a button in this section, click it to run your analysis. Since the model card is intended to be shareable, you cannot use data stored within a personal CASUSER library.    Once the analysis is complete, you should see a summary courtesy of SAS Information Catalog. This summary contains the number of columns, number or rows, size, status, completeness of data, information privacy classifications, data tags, and data descriptions. If there are gaps in the description, tags, and status, then these can be corrected in SAS Information Catalog. Working with data in SAS Information Catalog may require advanced permissions, so work with your data engineers or data owners to ensure your data metadata is complete. Model card users can click the name of the dataset within this section and open the data within Information Catalog in a new tab and make changes quickly if they have the appropriate permissions. Having complete data about your data is a best practice for building trustworthy models!    Updating the Model Card Data Summary section Model Summary   The Model Summary section examines the model’s performance at the time of training, which corresponds to the training donut charts in the Overview tab. The information in this tab is populated automatically when a model is registered from SAS Model Studio. The Model Summary section includes information about the model target, algorithm, development tool and version, various accuracy measures across training, testing, and validation splits, generalizability, and variable importance. If you’ve selected a variable in SAS Model Studio to assess for bias, those metrics will also appear in this tab. Overall, this tab provides a ton of information about how well the model was performing during training which can provide a baseline for monitoring model performance over time.      Complete Model Card Summary section   Model Audit   While the Model Summary section focuses on the model’s performance at the time of training, the Model Audit section reports on performance over time.  The Model Audit section also provides a deeper dive into the performance monitoring donut charts in the Overview section.   The Model Audit section relies on two capabilities of SAS Model Manager: performance monitoring and Key Performance Indicator (KPI) alert rules. Performance monitoring reviews model performance over time in batch at user-defined time points. To create a performance monitoring report, you need data with the actual or ground-truth values, and SAS Model Manager graphs model performance over time. Without the ground-truth, you will only get metrics on data drift. You can create a performance monitoring report in a few moments from the project, as outlined in these steps or the demo video below:   Creating and running a performance monitoring definition    Next, you will set your KPI thresholds in the project properties under model evaluation. You can create a rule in just a few clicks, as outlined in the documentation or the quick demo below:   Creating KPI alert rules   Now, you can view the latest model accuracy, fairness, and model drift metrics against your thresholds on the latest run of performance monitoring.      Complete Model Card Model Audit section   Pulling it All Together   This demo video walks through creating a model and a complete model card:   Creating a complete Model Card   Now you should have everything you need to create a complete nutrition label for your model! Stay tuned for the next article about creating a model card for a Python model. Otherwise post your questions, feedback, and ideas in the comments below!   ... View more

SAS Model Manager supports multiple publishing destinations so models can be deployed in the location and form that best fit their use case. For example, SAS Model Manager can deploy models to batch, real-time, container, and database locations. In a previous article, I discussed the CAS destination and how to score a model deployed to CAS over REST API. SAS Cloud Analytic Services provides the run-time environment for data management and analytics on SAS Viya. Thus, you will hear CAS referenced as a place to store data, but also as an analytical engine. CAS operates in-memory to provide the best performance for big data. And of course, models deployed to CAS can be scored using SAS code. In this article, we'll review the steps required to deploy a model into CAS and score new data using that model in SAS code.    Step 1: Create the CAS destination   While a CAS destination may not be available out-of-the-box, an admin can create one in just a few steps using the SAS Viya CLI or from Environment Manager. From Environment Manager in SAS Viya, first click Publishing Destinations. Then click the folder icon in the upper-right corner to create a new destination. Select the type as CAS, give the destination a name, select the CAS server, select the CAS library, and finally give the model table a name before clicking save.     Step 2: Publish to CAS   Now that we have configured our destination, we are ready to publish our model. We can deploy a SAS, Python, or R model into CAS. First, select your model(s) and hit the Publish button. From the Publish Models dialogue, select the CAS destination. Give your published model a memorable name before clicking the Publish button.   Now we are ready to open a new SAS program in SAS Studio and start using our deployed model.   Step 3: Start a CAS session   To interact with CAS, we need to start a session. We can start a CAS session and access the available CAS libraries by running the following code: cas; caslib _all_ assign;   Step 4: Load data into CAS   If your data isn't already in CAS, there are several methods for loading data into CAS. Using the Data Explorer on SAS Viya, under Manage Data in the main menu, we can build connections to a database or import a local file in just a few steps. We can also load data programmatically, with methods supporting data from various locations.   /*****************************************************************************/ /* Load file from a client location ("pathToClientFile") into the specified */ /* caslib ("myCaslib") and save it as "tableNameForLoadedFile". */ /*****************************************************************************/ proc casutil; load file="pathToClientFile" outcaslib="myCaslib" casout="tableNameForLoadedFile"; run; /*****************************************************************************/ /* Load SAS data set from a Base engine library (library.tableName) into */ /* the specified caslib ("myCaslib") and save as "targetTableName". */ /*****************************************************************************/ proc casutil; load data=library.tablename outcaslib="myCaslib" casout="targetTableName"; run; /*****************************************************************************/ /* Load a table ("sourceTableName") from the specified caslib */ /* ("sourceCaslib") to the target Caslib ("targetCaslib") and save it as */ /* "targetTableName". */ /*****************************************************************************/ proc casutil; load casdata="sourceTableName" incaslib="sourceCaslib" outcaslib="targetCaslib" casout="targetTableName"; run;   Step 5: Score data   Now to call the model, we will use the runModelLocal action. You will need a few parameters to fill out the code below, including: modelName: the name you gave your model when you published it in step 2. modelTable: the name of the table and associated CASLib for the destination created in step 1. inTable: the name of the table you want to score using your model. outTable: a name for the scored table. proc cas; loadactionset "modelPublishing"; runModelLocal / modelName="deployed_name", modelTable={caslib="destination_caslib", name="destination_table_name"}, inTable={caslib="sourceCaslib", name="sourceTableName"}, outTable={caslib="targetCaslib", name="targetTableName"}; run; quit;   Step 6: Promote data   Our table was scored within our CAS session and as such, is not available outside our session. But, with one more call we can promote our table, so it is available globally, even after our session is gone. Update the code with your library and table names. proc casutil; promote casdata="sourceTableName" incaslib="sourceCaslib" outcaslib="targetCaslib" casout="targetTableName"; run;   Step 7: Close session   Our session will expire after a set amount of time passes, but we can be good stewards of our resources by closing our session once we are done. cas casout terminate;   Step 8: Integrate   From here, you can use the scored data or the scoring code within a process that supports your use case best. For example, the scored data can be used within a dashboard or report in SAS Visual Analytics. Or you can take the code into a SAS Studio flow to string it together with other processing steps. And if you need the code scheduled, you can create a job around the code and schedule it. There are several options to fit your use case and needs to create a seamless analytics experience!   What to learn more about model execution in SAS Model Manager? Then check out these resources: Model Execution in SAS Model Manager Scoring using Models Deployed into CAS over REST API: A Step-by-Step Guide Scoring using Models Deployed into MAS over REST API: A Step-by-Step Guide ModelOps and DecisionOps for the Modern World What's New with SAS Model Manager? SAS Runtime Container   ... View more

Azure OpenAI Service provides access to Large Language Models (LLMs) through REST API, their Python SDK, and a web-based interface in the Azure OpenAI studio. Various models are available out of the box, including GPT-4, GPT-4 Turbo with Vision, GPT-3.5-Turbo, and Embeddings model series. You can read more about Azure OpenAI Service here. To incorporate these models in SAS Intelligent Decisioning and SAS Model Manager, we just need to create a Python function and scoring code calling the REST API of a model deployed in Azure OpenAI. At the time of writing this notebook, the Python SDK does not support all calls to each model type, but that may change in the future.   Before you can leverage the model, you must create an Azure OpenAI Service resource and deploy your model. This tutorial walks through setting up an Azure OpenAI Service and deploying a model in a few simple steps. For this example, I’ve used GPT-3.5-Turbo, but not all models are available in all regions. Next, I recommend walking through this tutorial with REST as your preferred method. Importantly, save your endpoint, API-key, and deployment name. We will need them for the next steps.   I’ll outline the steps below, but the code for this example is available in this notebook.   Integration with SAS Model Manager   To run our model in SAS Model Manager, we need to write our score code and save it as a .py file. We also need to specify the inputs to our score code, the outputs the score code generates, and properties of the model. Then we can register it all into SAS Model Manager directly from our notebook using the python-sasctl package.   For our score code, we will need the following: An import statement for the requests package. The requests package allows us to call REST APIs from Python. A function definition. When defining our function, we must consider how we want users to interact with the model. Will they be able to pass their prompt directly to the function? Will they pass parameters that populate a prompt template? It is your function! Build it to suit your needs. An output statement so SAS Model Manager knows what output to expect. Within your function definition, you can add any pre-processing of your prompt before saving it into your data json object. Using a LLM may require a key, so select a key management strategy that works best for your use cases and security needs within the function definition. The example notebook demonstrates a few options for key management.   Within your function, make the POST request using your model’s URL, passing along the key and data json object, and save the output of the POST request to a variable.  Optionally, you can add any post-processing to the output of the REST API call before returning the output.   Thus, your score code may looking something like:   def score(prompt): 'Output:answer_llm,finish_reason' url = 'YOUR-MODEL-ENDPOINT' k = 'YOUR-API-KEY' h = {"Accept": "application/json", "Content-type": "application/json; charset=utf-8", "api-key": k} data = {"messages":[{"role": "user", "content": prompt}]} response = requests.post(url, json = data , headers = h ) jsonResponse = response.json() finish_reason = jsonResponse["choices"][0]['finish_reason'] answer_llm= jsonResponse["choices"][0]['message']['content'] return answer_llm,finish_reason   If you are working in a notebook, you can write the score code out to a file by adding this line to the top of your score code:   %%writefile YOUR-MODEL-NAME.py   Using the python-sasctl package, you can also generate the metadata for the model and register it to SAS Model Manager without needing to leave your Python development environment. After authenticating to the SAS Viya environment using python-sasctl, specify the input and output variables, create the model, and upload your score code file.   # Update these variables to match your project project = '[INSERT-YOUR-PROJECT-NAME]' model_name = '[INSERT-YOUR-MODEL-NAME]' algorithm = '[INSERT-YOUR-LLM-ALGORITHM]' # Specify input variables and output variables inputvariables = [{'name': 'prompt', 'role': 'input', 'type': 'string', 'level': 'nominal', 'length': 500}] outputvariables = [{'name': output, 'role': 'output', 'type': 'string', 'level': 'nominal', 'length': 500}] # Create the model model = mr.create_model( model=model_name, project=project, algorithm=algorithm, modeler=username, tool='Python 3', function = "Text Generation", score_code_type = 'Python', input_variables = inputvariables, output_variables = outputvariables ) # Add score code scorefile = mr.add_model_content( model, open('[INSERT-YOUR-MODEL-NAME].py', 'rb'), name='[INSERT-YOUR-MODEL-NAME].py', role='score' )   Now, check your SAS Model Manager project and you have executable score code for your OpenAI model that can be run in a scoring test or deployed to MAS or container destinations!   Integration with SAS Intelligent Decisioning   We can use the Python model we just developed and registered to SAS Model Manager within our decision flow as a model. But, if you don’t have SAS Model Manager, no need to worry! We can leverage a Python code file in SAS Intelligent Decisioning instead. To run the model, we need to create an execute function that we will then copy and paste the code into the Python code files in SAS Intelligent Decisioning.   This function will look really similar to the function we wrote for SAS Model Manager. The difference being that we must add a line to note the requests packages as a Dependent Package.   import requests def execute (prompt): 'Output: answer_llm,finish_reason' 'DependentPackages: requests' url = '-YOUR-MODEL-ENDPOINT' k = 'YOUR-API-KEY' h = {"Accept": "application/json", "Content-type": "application/json; charset=utf-8", "api-key": k} data = {"messages":[{"role": "user", "content": prompt}]} response = requests.post(url, json = data , headers = h ) jsonResponse = response.json() finish_reason = jsonResponse["choices"][0]['finish_reason'] answer_llm = jsonResponse["choices"][0]['message']['content'] return answer_llm,   Now you can leverage your LLM within a decision flow. This is great for building decisioning processes that can use a LLM to generate text while incorporating rules, various data points, and other machine learning models.   Conclusion   Now, you are all set to leverage a LLM deployed in Azure OpenAI in SAS Model Manager or SAS Intelligent Decisioning where it can be managed with other models in your organizations, combined with business logic, orchestrated with other models, and deployed into destinations within SAS Viya or beyond using containers.   Want to learn more about LLMs & Generative AI and SAS Viya? Then check out these resources! Understanding LLMOps: Navigating the waters of large language models Using a LLM-based digital assistant for SAS Optimization How to manage and govern prompts of large language models with SAS 5 steps to improve information extraction using trustworthy generative AI How to Create Your Custom LangChain Agent for SAS Viya Expanded SAS Viya uses generative AI to accelerate customer productivity SAS Viya helps businesses put GenAI models to work faster through robust LLM orchestration. ... View more

  Registration for the SAS Hackathon is open! Everyone needs to know about the SAS Hackathon because this is your opportunity to solve a real-world problem. Participants in the SAS Hackathon bring a problem, build a team, find data, and get to work. But they aren’t alone. As a part of the SAS Hackathon, teams have access to the breath of SAS Viya tooling for a month at no-cost. That’s right, if you participate in the SAS Hackathon, you get your hands on the most productive end-to-end data, analytics, and AI platform for free. This includes tools for: Accessing, preparing, and governing data Building robust visualizations, dashboards, and reports Training machine learning models in predictive analytics, computer vision, text analytics, forecasting, and optimization Managing and deploying models to real-time endpoints Creating automated and standardized decisioning flows   SAS Viya provides everything you need to build a complete analytics system at your fingertips. And not only is the environment and tooling free, but SAS is also providing training and mentoring at no cost.     Why would we do that? Well, we believe that analytics can change the world for the better. And we believe in our platform and its ability to improve outcomes using analytics. SAS Viya allows teams to move beyond science experiments to production decisioning.   Just ask the winners of the 2023 SAS Hackathon, Team JaWaRA (Jakarta Water Resources Analytics) who used the AI, machine learning, and streaming analytics capabilities from SAS Viya to build an Early Warning System as a part of their citywide Flood Control system. Jakarta, Indonesia’s capital and home to 11 million people, sits on wetlands laced by 13 major rivers and borders the Java Sea. More than 40% of its land is below sea level, and groundwater extraction has led to the ground sinking. Combine those conditions with stronger monsoons and higher tides caused by climate change, and the flooding threat is very real. As a part of the 2023 SAS Hackathon, Team JaWaRA analyzed real-time data on rainfall, water levels and river flow to better predict and mitigate flooding.   So, in addition to saving lives like Team JaWaRA, what can you do with SAS Viya during the Hackathon? You can: Train and deploy Python models to a real-time REST API endpoint that can be invoked from an application. Build an interactive dashboard that shows how small tweaks affect which actions are most optimal. Create a deployable decision flow that drafts and sends an email to a customer based upon their past behavior and likelihood to purchase. Incorporate Trustworthy AI into your analytical processing using the Trustworthy AI Lifecycle Workflow, Bias Assessment and Mitigation, Model Interpretability Techniques, and the upcoming model card (stay tuned to learn more)! Everything you generate is yours to keep after the SAS Hackathon in the hopes that you can continue to productionalize your work! After participating in the previous SAS Hackathon, 95% of participants would recommend SAS Viya and the SAS Hackathon! Ready to participate? SAS is here to help you along the way, but now is the time to pick your problem, build your team, and plan your approach.    The SAS Hackathon starts on September 16 th, but you need to register before August 30 th to participate! What to learn more? Then check out these resources: SAS Hackathon Website SAS Hackathon 2024 Kickoff Event SAS Hacker’s Hub Community SAS Hackathon Ideation: The Board Game ... View more

One of the key capabilities of SAS Model Manager is the ability to execute SAS, Python, and R models, so that these models can be used to easily score new data. In fact, SAS Model Manager supports multiple execution environments, from batch execution within SAS Viya to transactional REST-API endpoints hosted on SAS Viya or outside of it via containers.   Understanding these execution environments and their interactions with open source is a helpful skill for debugging and resolving issues in a timely manner. Today’s article will focus on the three most popular execution engines within SAS Model Manager with tips for administrators for addressing common issues.   SAS Cloud Analytics Service   SAS Cloud Analytic Services provides the run-time environment for data management and analytics on SAS Viya. Thus, you will hear CAS referenced as a place to store data, but also as an analytical engine. CAS operates in-memory to provide the best performance for big data. CAS supports execution of SAS, Python, and R models.   Within SAS Model Manager, CAS acts as both a publishing destination and the execution backing the Scoring Tests. Scoring tests provide a quick way to execute a model against a table within CAS. To create a new Scoring Test in CAS, a user selects a model, a version of that model, input data set, and an output library.   Executing models within CAS is just a few clicks for the user, but SAS Model Manager is performing several steps behind the scenes. SAS Model Manager will wrap the modeling code in a language called DS2. DS2 offers several advantages including parallel computation even within CPU-bound processes. This code includes steps to read data from CAS, execute the model code, and write the resulting data back to CAS. For Python models, an interface between DS2 and Python, called PyMAS, is invoked within the DS2 code. PyMAS handles the movement of data into and out of the Python model score code. For R models, SAS Model Manager automatically generates a Python wrapper around the R score code with the R model embedded within a Python process using the rpy2 package. This means that PyMAS can also act as the intermediatory between the DS2 code and the wrapped R model.   Tip: When executing Python and R models, do you notice pypgm, resultCode, and revision as columns in the output table? These are variables used by SAS Model Manager within the DS2 code for handling score code and results codes. Each PyMAS package instance represents exactly one Python module revision, and you may only see a value populated in this column whenever a new revision is invoked. If your Python code fails, a -1 is returned for the revision.  Furthermore, a value of 0 for resultCode indicates successful execution for that row of data, but a non-zero value indicates a failure. A list of resultCode meanings can be found here.   SAS Micro Analytic Service   SAS Micro Analytic Service (MAS) is a stateless, memory-resident, high-performance program execution service that is available as an out-of-the-box publishing destination in SAS Model Manager. The MAS destination hosts the model on SAS Viya, which reduces the amount of configuration work required to score new data using the model. Additionally, the MAS destination exposes a REST API endpoint to the model, making it easy to incorporate the model into a wider process or application. MAS is best suited for ad-hoc transactional scoring in near-real time. For example, MAS is a great fit for determining if a credit card purchase is fraudulent right after the card is swiped. MAS is also great for scoring data in an application right after the applicant clicks submit.   We can deploy a SAS or Python model into MAS in just a few clicks from the SAS Model Manager user interface. First, select your model(s) and hit the Publish button. From the Publish Models dialogue, select the MAS destination. Give your published model a memorable name before clicking the Publish button. In just a few seconds, we are ready to start scoring data over REST API using our model!     Just like model execution in CAS, SAS Model Manager automatically wraps in the scoring code in DS2 and leverages PyMAS as the interface between DS2 and Python. But within MAS, a transaction is scored via a POST request with the input data as a JSON payload. This means that input data to score is sent to the deployed model via REST API and the resulting output is returned as a part of that same call.   SAS Container Runtime   Models can also be deployed outside of SAS Viya too. SAS Model Manager can build containers for SAS, Python, and R models that are published to container registries in Azure, Docker, GCP, and AWS. These containers are light-weight, portable, standardized, and scalable. Using SAS Container Runtime, you can easily take advantage of the cloud infrastructure, or local installation, to deploy applications with small footprint and are highly scalable and highly available. This ensures resources can be fully utilized to quickly execute the largest number of models and reduce the effort that is required to manage the environment.   Upon publishing, SAS Model Manager will build the container, install dependencies, add the scoring code, and push the container to the specified registry. SAS Model Manager leverages different base containers, depending on the score code type of the model. This allows us to minimize the container footprint by only including what is necessary.   After deploying the container into Kubernetes or Docker, users can score data via REST API with a JSON payload, just like with MAS. The difference is that containers provide scalability and the ability to execute without SAS Viya. Installation and Configuration   When installing SAS Viya, there a few key steps to ensure models run smoothly within SAS Viya. First, ensure that the ASTORES persistent storage volume (PV) and the overlays are configured during the deployment of SAS Viya. The overlays mount the directory paths that point to the ASTORES persistent volume claim (PVC).   Next, for running Python models, also ensure that a the customer-prepared Python installation (and any required packages) Kubernetes persistent volume (PV) and overlays are configured during the deployment of SAS Viya. Additionally, the PyMAS package with the environment variables pointing to the python executable file and mas2py.py file should be configured.  There are also minimum versioning and packaging requirements for the Python environment.   Finally, for running R models, complete the steps for Python, including set up for rpy2, and ensure that a the customer-prepared R installation (and any required packages) Kubernetes persistent volume (PV) and overlays are configured during the deployment of SAS Viya.   Tip: When working with CAS and MAS destinations, take note that only one Python and one R environment is supported. The Python and R language as well as their many packages and libraries are not necessarily designed to play nicely across versions. If you are seeing issues when unloading and calling your model, there may be a version mismatch in model decencies. You can mitigate these issues when deploying to a container by leveraging the requirements.json file.   Administration   Running open-source code can introduce security risks. Thus, running open-source code in SAS Model Manager through the scoring test or performance monitoring requires escalated privileges. Folks with approval to run open-source code should be added by an Administrator to the CASHOSTACCOUNTREQUIRED group.   Before users can publish models to a destination, it must be created by an administrator. A MAS destination is often available out-of-the-box, and an admin can quickly create a CAS destination in Environment Manager. For container destinations, I recommend leveraging the Command Line Interface. There are examples for creating each destination in GitHub.   Code Requirements   To run Python and R models within SAS Viya, the score code files must follow a specific format. Data scientists can find the score code format required for Python models and R models in the SAS Model Manager documentation. This score code must include import statements for any necessary modules or library, a score function definition, and an output statement. Packages like Python-sasctl and R-sasctl can also help generate the correctly formatted model score code.   Want to learn more? Check out these resources: Scoring using Models Deployed into CAS over REST API: A Step-by-Step Guide Scoring using Models Deployed into MAS over REST API: A Step-by-Step Guide ModelOps and DecisionOps for the Modern World What's New with SAS Model Manager? SAS Runtime Container   Did I miss any helpful tips? Add them in the comments below!  ... View more

SAS Innovate is the data and AI experience for business leaders, technical users and SAS partners. Registration is  open for SAS Innovate, which will take place in Las Vegas April 16 th - 19 th .  Attendees get to hear from top voices in the field, including Organizational psychologist Adam Grant, the former chief Decision Scientist at Google and CEO at Data Scientific Cassie Kozyrko, tech reporter Kara Swisher, chair of The National AI Advisory Committee Miriam Vogel, and more. Additionally, there will be more than 200 sessions, networking opportunities, and hands-on boot camps. This is an event you do not want to miss!     Let’s dive into all things ModelOps at SAS Innovate. In this article, we will go over the sessions, demos, and events where you can learn how to improve outcomes through better analytically driven decisions and model management.   Participate in a SAS Hackathon Boot Camp The annual SAS Hackathon is a month long-event where you bring a problem and your best problem-solvers, and SAS provides the tooling, compute, enablement, and mentoring, all at no charge.  The SAS Hackathon starts in September this year, but you can still get hands-on with SAS tooling at the SAS Hackathon Boot Camp! Participation in the Boot Camp is complementary for all SAS Innovate attendees, but be sure to select which Boot Camp option you would like to attend while registering.      Model Registration for SAS® Viya® Programmers Made Easy Our first suggested session is for our SAS modelers. The process of building AI models does not end once the model has been trained. Join this session to learn about the new REGISTERMODEL procedure in SAS Viya that makes it easy to register a model from a Viya programming environment to SAS Model Manager.   The Easy Button: One Click to Native Open Source Model Execution in SAS® Viya® 4 Our next suggested session is for our open-source modelers. Integrating complex open source models into your business operations has never been easier! Many open source libraries provide containerized execution environments that can easily be deployed alongside Viya 4 with the push of a button. Learn how to get the most out of Viya by executing models natively with PyTorch, TensorFlow, XGBoost, H2O and more.     Integrating SAS® Intelligent Decisioning and LLMs for Automated Analytical Decisioning How can you combine advanced analytics, Natural Language Processing (NLP) and decision orchestration to deliver exceptional customer experience and make analytically driven decisions at scale? Join this session to learn about the cutting-edge integration between SAS Intelligent Decisioning and large language models (LLMs), showcasing how AI and NLP can work together to drive real-time interactions.   Asset Monitoring Anomaly Detection at Scale Using Vibration Sensors Georgia-Pacific had a problem: unplanned equipment outages were costing the company money and disrupting operations. Then they started using vibration sensors for predictive maintenance to enable asset monitoring to harness the data they produced and transform it into actionable intelligence. Learn about how Georgia-Pacific implemented the end-to-end process with SAS® to analyze this vibration data and generate alerts on hundreds of assets with thousands of models in a distributed environment.     Navigating the Ethical Landscape of Generative AI Generative AI has the potential to revolutionize the way we write, develop, create, and solve problems. However, like any other system, it can introduce ethical risks if not deployed responsibly. Executives must understand the advantages and disadvantages of generative AI to guide their organization towards successful implementation. This session will equip the audience with key questions and considerations for conducting an ethical inquiry into generative AI from an executive perspective.   Enforcing Trustworthy AI Standards in Analytical Projects SAS Communities readers may be familiar with our release of the Trustworthy AI Life Cycle Workflow earlier this year. Now join yours truly as we how show SAS tooling makes NIST recommendations easier to adopt and how to ensure that analytical models don’t create disparate impact or harm. Attendees to this session will gain a greater understanding of trustworthy AI best practices as well as user roles and expectations for building responsible analytics. We'll also demonstrate how SAS supports a production model with documentation showing that an organization has done its due diligence for a fair model with processes that don’t cause undue harm.     LLMOps With SAS® Model Manager Can’t get enough of LLMs? Neither can we! Delve into the critical aspects of managing and operationalizing LLM models effectively. In this session, we will explore key topics such as prompt versioning, testing, output comparison and red teaming to ensure that your machine learning operations are efficient, secure and adhere to your company brand guidelines.   Be the Conductor With Process Orchestration for SAS® Viya® Need to orchestrate complex ModelOps process between multiple tools? Then we have the session for you! Using Apache Airflow as the flow execution engine, this new feature enables users to craft and manage custom flows across the apps in Viya. Offering robust access controls, dynamic partitioning of SAS tasks and nested sub-flows, process orchestration seamlessly integrates with the APIs in Viya to support complex business processes.   Bending the Rules: How to Use Linguistic Models to Improve the Performance of LLMs We have another great session on LLMs! Linguistic rules can be used to curate data and aid in training, fine-tuning, and prompt-tuning, as well as evaluating the results generated by LLMs. By combining linguistic rule-based models with LLMs, we can help to improve the quality and accuracy of LLMs and enable them to perform better on various tasks while cutting costs.     AI Governance: From Compliance to Strategic Differentiator Every disruption creates opportunities for organizations to emerge as new thought leaders. The impending release of the European Union AI Act spurred PZU, Poland’s largest insurance company, to prepare the enterprise for compliance and beyond. We'll tell you about PZU’s initial successes and lessons learned as it transitioned to modern, future-proof AI governance in collaboration with SAS’ Data Ethics Practice. Hear practical tips for strategies, how to make AI governance a strategic differentiator and how SAS resources can help with your governance journey.   LLM ModelOps in Action: A Guide to Using SAS® Model Manager for Large Language Models Last session on LLMs (I promise). In the rapidly evolving landscape of AI, large language models (LLMs) have revolutionized the way we interact with technology, offering unprecedented capabilities in generating humanlike text. However, the true potential of these models can only be unleashed when they are efficiently deployed and managed. This demo introduces participants to SAS Model Manager, a comprehensive ModelOps solution designed to streamline the management, deployment and monitoring of a wide range of machine learning models. Through a live demonstration, attendees will learn how to leverage SAS Model Manager for containerizing, deploying and serving LLMs, ensuring users can harness the full power of AI innovations in their own projects.   Visit the ModelOps Booth in the Innovation Hub The Innovation Hub will feature dozens of booths covering data management, machine learning, visualization, industry solutions, programming and more. The ModelOps booth will be staffed with experts from teams across SAS that are ready to answer your questions, hear your feedback, and provide product demos. The ModelOps booth will be in the SAS Spotlight area of the Innovation Hub, next to the Decisioning booth. The Innovation Hub is open for most of the conference, but check the Agenda at a glance for the most up-to-date hours.       Ready to Learn More? Whew, that is a lot of great sessions! Which one are you most excited for? Did I miss a must-see session? Let me know in the comments below!   Register for SAS Innovate now (if you haven’t already) to get the best deals! Want to learn more about what SAS Innovate has to offer? Then check out these resources: SAS Innovate Home Page 5 reasons to participate in the SAS Hackathon boot camps 7 sessions you don’t want to miss at SAS Innovate What’s happening at the SAS Innovation Hub? Unveiling the marvels of SAS Innovate: How it all comes together   ... View more

SAS has just released an experimental version of our Trustworthy AI Life Cycle Workflow for use with SAS® Model Manager and SAS® Workflow Manager on SAS Viya 2024.01 and later. Our Trustworthy AI Life Cycle workflow enforces standards and best practices set by the AI Risk Management Framework defined by the National Institute of Standards and Technology (NIST). The workflow allows organizations to document their considerations of AI systems’ impact on human lives.  Our workflow includes steps to ensure that the training data is representative of the population that is impacted, as well as that the model predictions and performance are similar across protected classes These steps help ensure that the model is not causing disparate impact or harm to a specific group. Furthermore, you can ensure that your model remains accurate over time by creating human-in-the-loop tasks to act when additional attention is needed.     The aim of the workflow is to make NIST’s recommendations easier to adopt for organizations by specifying individual roles and expectations, gathering required documentation, outlining factors for consideration, and leveraging automation to ease adoption. The result of the workflow is a production model with documentation to support the assertion that the organization has done its due diligence to provide evidence that the model is fair, and their processes do not cause harm.   Given the growing landscape of Responsible and Trustworthy AI, this workflow has been marked as experimental to gather your feedback. As you use the sample workflow, we would like to know what works well and what can be improved upon. Feedback can be provided in the Issues tab of our SAS Model Management Resources GitHub page.   To get started using this workflow, perform the following steps:   Perform first-time configuration steps for SAS Workflow Manager, if not already configured. Download the sample workflow definition from the SAS Model Management Resources GitHub page. Import and activate the workflow definition using SAS Workflow Manager. Workflow administrators should review timer and role defaults to ensure the values match their organizational processes. When a modeling project is available within SAS Model Manager, you can start the workflow. You will be prompted to specify the model owner. The workflow consists of user tasks, which will appear within the Task category in SAS Model Manager. Tasks only appear for the users specified in the task role to ensure that right individuals are providing documentation and approvals. The first task will be for the Model Owner in the Tasks category. Note: Users might need to refresh the tab to see new tasks. Several of the tasks involve providing documentation, aligning with NIST recommendations. To help compile this documentation, a companion template document in provided in the SAS Model Management Resources GitHub page, so be sure to download it and store it among the model files.   This workflow has several pieces to support the life cycle of a model. Let’s dive into each section.   Identify Stakeholders   Analytics is a team sport with data scientists, IT/engineering, risk analysts, data engineers, business representatives and other groups all coming together to build an AI system. Once the model owner is identified, they must then specify the other project roles. This workflow supports both single users and groups in the stakeholder roles. Additionally, users or groups can be specified for multiple roles. The roles used in the workflow are:   Model Owner – the decision maker for the analytics project. They will be responsible for documenting the purpose of the project and will provide approval or feedback as each piece of the project is assessed for Trustworthy AI practices. Model Developer – the data scientist(s) that develops the model. Model Engineer – the engineering or IT resource(s) that deploys the model. Model Risk Owner – the risk analyst(s) or risk manager(s) that manages and document the model risks. Data Engineer – the engineering resource(s) that prepares the data for modeling. Domain Expert – a resource that is available to address business or domain questions.   Verify Project   This section consists of mostly system tasks to confirm that the project metadata has been populated and that models exist within the project. If information or models are missing, users are promoted to correct the problem. Consider this section a completion check prior to advancing further.   Documents Project   The stakeholders listed previously are responsible for documenting various pieces of the project in alignment with NIST’s AI Risk Management Framework. In this section, responsible parties will be prompted to provide documentation via the tasks in SAS Model Manager. To aid in this compilation, we recommend using the provided template.   The documentation compiled in this step examines the proposed model usage, end users, expected performance of the AI system, strategies for resolving issues, potential negative impacts, deployment strategies, data limitations, potential privacy concerns, testing strategies, and more. Users can add additional documentation to cover any additional needs or use cases not covered by the workflow. By compiling this documentation, the various stakeholders can align on potential risks posed by the project as well as confirm the project plan.      Assess Data   The section for assessing data addresses two key factors: privacy and bias. Data privacy and bias are not concerns for all analytical projects. For example, some machine predictive maintenance projects rely on telemetry data and will not include information about individuals. For these cases, there is an option to by-pass the data privacy and bias questions.    For use-cases where Personally Identifiable Information (PII) is not required, the workflow prompts the data engineer to remove or mask PII. Otherwise, the risks of leveraging PII for modeling must be documented. Keeping PII data within the analytical base table increases risk of private information being leaked to bad actors.   Next, the workflow examines potential data biases, such as the inclusion of protected class variables or proxy variables for modeling as well as the representativeness of the data. Potential bias risks must be addressed or documented before moving forward. Using training data that is not representative of the target population can create a model that is less accurate for specific groups, resulting in undue harm for those groups. Including proxy variables or protected class variables in the training data can create a model that treats groups differently. For some use cases, such as health care, these variables may be important predictors for disease risk. For others, such as models aimed to predict who should receive a service or a benefit, this can lead to discrimination.     Assess Model   Models are assessed against three factors: performance, fairness, and explainability. Performance data provides information on how well the model can predict the desired event, which helps us understand how useful the model may be for our AI system. For gathering model performance metrics on training, testing, and validation sets, SAS provides functionality in support of several modeling types. For models developed in the graphical pipelines of SAS Model Studio, this information is calculated automatically and will be registered with the model. For models developed programmatically in SAS, we recommend providing the scored data to the macro at registration time. Similarly, this information can be calculated using the scored data using functions provided in the python-sasctl and r-sasctl open-source packages for python and R models.   For models that require an evaluation of potential bias, the model developer is prompted to calculate and provide the model’s performance and average prediction for each group of the protected class.  A model that performs worse on specific groups may point to problems with a lack of representation within the training data. Additionally, a model that is less accurate on specific groups may cause disparate impact or harm to those groups. Depending on the use case, a model that predicts different outcomes based on those groups may also cause undue harm. For models developed in SAS Model Studio, data scientists can select which variables to asses for bias in just a few clicks. Outside of SAS Model Studio, users can leverage the Assess Bias Action Set. Python-sasctl provides a function wrapping this action set for Python developers. When bias is detected in a model, model developers can leverage the Mitigate Bias Action Set to help mitigate model bias.   The last piece models are assessed against is model explainability. Model explainability helps stakeholders understand how the inputs to a model affect the model’s prediction. By understanding the relationship between the inputs and prediction, data scientists and domain experts can determine if the model captures well-known connections between factors. Additionally, explainability can provide evidence that a model is fair by demonstrating that a model is using factors that are unrelated to protected group status. Some models lend themselves to easy explainability. For example, the form of regressions and decision trees directly define how input variables are used to calculate the model prediction. Other models, such as gradient boosting or random forest, are hard to interpret and require additional methods to explain the model. There are several model interpretability techniques available including:   Local Interpretable Model-Agnostic Explanations (LIME) Partial Dependence (PD) Kernel Shapley Additive Explanations (Kernal SHAP) Individual Conditional Expectation (ICE) Hyperparameterized Shapley value estimation (HyperSHAP)   In SAS Model Studio, users can specify to run model interpretability techniques in the Post Training Properties of the Supervised Machine Learning nodes. For programming based models, users can explore the Explain Model Action Set.     Select Champion Model   The champion model is the best candidate for production. Users can mark a single model as the champion model but can mark multiple challenger models within the project in SAS Model Manager. This section of the workflow checks if a champion model exists, and if not, prompts the Model Owner to select a champion model before moving forward. When selecting a champion model, it is important to keep in mind performance and fairness assessment results from the previous steps. Additionally, the Model Owner should ensure the model qualifies based on qualities like the appropriateness of variables, explainability, and others. This model will undergoing further testing and approval in the next step before deployment to production.   Test and deploy model   Prior to production, the champion model should be tested using the framework specified in the project documentation. These tests may involve ensuring that the model can score data within SAS Viya and can score data within the publishing destination. Models should also be tested against unseen values and unseen data sets, such as a testing or validation set, to confirm performance continues to meet expectations.   After a model is tested and approved for production, it can be deployed for production. To prevent a model from going to production before approvals are attained, we recommend leveraging permissions within SAS Viya to prohibit user publishing. Additionally, administrators may need to do one-time configuration of their publishing destinations if they are not using an out-of-the-box destination. SAS Model Manager supports a variety of destinations, including a REST API endpoint hosted within SAS Viya, called MAS. SAS Model Manager also supports containerizing models, which also leverage a REST API endpoint, but will be hosted outside of SAS Viya. Other destinations include CAS and select databases.   Once the model is deployed, the model engineer will need to ensure that the model is incorporated in their business processes, whether that be using the REST API call within their web applications or creating a dashboard to review the results of batch scoring.     Monitor model and review project   All models decay, but models don’t decay at the same rate. Model decay leads to a decrease in model accuracy, so any decision being made using that model will be wrong more often. SAS Model Manager supports Performance Monitoring to pinpoint when decay begins to occur. Performance Monitoring can be combined with Key Performance Indicators (KPIs) to create thresholds for alerting. For this section, data scientists or engineers should ensure project performance monitoring and KPIs are defined. When model performance no longer meets the predefined threshold, the model owner must decide which step to take next, whether it be to retrain the model, select a new champion model, end the workflow, or retire the project.      Recurring Audit   The NIST AI Risk Management Framework recommends reviewing the project on a recurring basis to validate that the project parameters, assumptions, stakeholders, use case, data, and model are still valid. This provides a regular audit of AI Systems, allowing organizations to remove systems no longer in use. Retiring unused systems help save on cloud costs, reduces the number of models managed, and reduces the attack space for bad actors.     Once you have tried out and reviewed the workflow, we ask that you provide your feedback in the Issues tab of our SAS Model Management Resources GitHub page. Did this workflow fit well in your ModelOps processes and if not, what can we improve upon? Were the documentation and tests valuable to supporting your organization’s needs for Responsible AI? Are there factors that you would like to see considered for the next iteration? What additional can we provide to ease adoption? We will actively review provided feedback and incorporate what we can into the next iteration of this workflow!   Want to learn more about building Responsible AI applications? Check out the following resources:   A Comprehensive Approach to Trustworthy AI Governance Trustworthy AI Using SAS®: A Technical Discussion Trustworthy AI: 3 reasons we need it now How to Think About Trustworthy AI in 2024 and Beyond On-Demand Webinar Accountability in AI: A key ingredient for trust Inclusivity: A guiding principle for responsible innovation Data quality: The foundation for trustworthy AI The ethics of responsible innovation: Why transparency is key ... View more

SAS Model Manager supports both model and project versioning to help track changes and preserve model snapshots across important milestones. Versioning is a key capability that supports proper model governance and validation processes. For today’s article, let’s review how to use project and model versioning as well as discuss recent enhancements to versioning.   Project Versioning    Projects hold multiple models and the information specific to one analytical use case. Projects can be versioned within SAS Model Manager, but project versioning has a few key differences from model versioning. For example, each project version holds a distinct set of models. Commonly, a new project version is created whenever the training data for the set of models updates, such as during model retraining. In fact, whenever a user opts to retrain their modeling projects from SAS Model Manager, a new project version is generated automatically.   Project versions can be created, deleted, renamed, and viewed from the Version drop-down in the Models tab of the project. This short video demonstrates how to interact with project versions:     Model Versioning    Each version of a model holds a variety of files and information, including the score code, model ASTORE or pickle file, properties, link to the training table, requirements, and more.   From the Versions tab of the model, users can view each version of the model as well as create a new model version. This short video demonstrates how to interact with model versions:   As of writing just after the release of SAS Viya 2024.01, a new model version is generated by the following activities: Publishing a model. Setting a model as champion. Making changes to the original model within SAS Model Studio and re-registering the model. Manually creating a new model version.   Whenever a new model version is created, the previous model versions are locked from editing. Locking the previous model versions from editing ensures that the files and information we have within SAS Model Manager accurately reflects what is available in deployment destinations.   Users have provided several pieces of feedback for how we can improve versioning to better fit their needs. We have already released several new capabilities based upon that feedback and are currently working on addressing more of that feedback.  So next, let’s review what’s new and on the horizon for versioning within SAS Model Manager.   What’s New with Versioning in SAS Model Manager   Prior to our recent changes, users were restricted to only use the latest version of a model for decision flows, score testing, publishing, and performance monitoring. After discussions with our users, we decided to remove that restriction. Allowing users to leverage previous model versions provides greater flexibility in model validation and governance processes. Let’s dive into how you can start using each of these new capabilities.   Use Any Model Version within a Decision Flow   Since SAS Viya 2023.05, SAS Intelligent Decisioning users have been able to specify which version of a model is used within a decision flow. After a model is added to the flow, users can select the model version from the properties pane:     This allows users to create decision flows using a specific model version or opt to always pick up the latest version of a model. Multiple versions of a model can exist within the same decision flow, enabling A/B Testing of new model versions against previous ones.   Score Test All Versions of a Model   With the release of SAS Viya 2023.09, any version of a model can be selected for a score test. After users select their model inside a scoring test, they can then select which version of that model they would like to use.  Users have two options for selecting a model version to use for score testing: Select a model version by number: this option will create a scoring test that will always use the selected version of the model. Select to use latest model version: this option will always use the latest version of a model. If a new version of a model becomes available, re-running this test will use the newest version of a model. These two options give users a static option and a dynamic option for creating scoring tests.   Creating scoring tests for previous model versions allows users to continually test their production and pre-production models and provide stronger model validation over time.   To learn more about this new capability, see the following demo:     Publish All Versions of a Model   From SAS Viya 2023.10 onwards, any version of a model can be published. From the publishing dialogue, users can select which version of a model they would like to publish. This means that the same version of a model can be deployed in multiple destinations, such as a testing environment and a production environment. This also allows for easy roll-back of a production model to a previously deployed version.   To learn more about this new capability, see the following demo:     Monitor All Versions of a Model   Beginning on SAS Viya 2024.01, any version of a model can be monitored. Like score testing, users have two options for selecting a model version to monitor: Select a model version by number: this option will always monitor the selected version of the model. Select to use latest model version: this option will always monitor the latest version of a model. If a new version of a model becomes available, re-running the performance monitoring report will use the newest version of a model.   Monitoring the performance of a model across versions can help users react to decay if multiple versions of a model are deployed across destinations or determine if a production model should be rolled-back to a previous version.   To learn more about this new capability, see the following demo:     Upcoming Version Enhancements   Our users have reported that they would like automatic version incrementing (also know as version bumping) when publishing a model or marking a model as champion to be optional. This is the next piece of feedback we would like to address for versioning enhancements. It is necessary to continue to lock models from editing when publishing or marking a model as champion to ensure models within SAS Model Manager match what is available in external destinations and environments, but we can let the user decide when to create a new model version.   Versioning is a key component for robust model governance and management. Based on user feedback, we have made adjustments to better support their processes and use cases.  Interested in providing your own feedback to the SAS Model Manager team? SAS Product Suggestions is a board for product feedback and ideas.     Want to learn more about SAS Model Manager? Then check out the following resources: A Tour of SAS Model Manager Ask the Expert - How Do I Use SAS Model Manager?  Ask the Expert - Mastering ModelOps: Supercharge Your Analytics Journey  IDC MarketScape: Worldwide Machine Learning Operations Platforms 2022 Vendor Assessment ModelOps Explained: A Starter’s Guide to Deploying and Managing AI and Analytical Models ... View more