SAS Innovate is the data and AI experience for business leaders, technical users and SAS partners. Registration is  open for SAS Innovate, which will take place in Las Vegas April 16 th - 19 th .  Attendees get to hear from top voices in the field, including Organizational psychologist Adam Grant, the former chief Decision Scientist at Google and CEO at Data Scientific Cassie Kozyrko, tech reporter Kara Swisher, chair of The National AI Advisory Committee Miriam Vogel, and more. Additionally, there will be more than 200 sessions, networking opportunities, and hands-on boot camps. This is an event you do not want to miss!     Let’s dive into all things ModelOps at SAS Innovate. In this article, we will go over the sessions, demos, and events where you can learn how to improve outcomes through better analytically driven decisions and model management.   Participate in a SAS Hackathon Boot Camp The annual SAS Hackathon is a month long-event where you bring a problem and your best problem-solvers, and SAS provides the tooling, compute, enablement, and mentoring, all at no charge.  The SAS Hackathon starts in September this year, but you can still get hands-on with SAS tooling at the SAS Hackathon Boot Camp! Participation in the Boot Camp is complementary for all SAS Innovate attendees, but be sure to select which Boot Camp option you would like to attend while registering.      Model Registration for SAS® Viya® Programmers Made Easy Our first suggested session is for our SAS modelers. The process of building AI models does not end once the model has been trained. Join this session to learn about the new REGISTERMODEL procedure in SAS Viya that makes it easy to register a model from a Viya programming environment to SAS Model Manager.   The Easy Button: One Click to Native Open Source Model Execution in SAS® Viya® 4 Our next suggested session is for our open-source modelers. Integrating complex open source models into your business operations has never been easier! Many open source libraries provide containerized execution environments that can easily be deployed alongside Viya 4 with the push of a button. Learn how to get the most out of Viya by executing models natively with PyTorch, TensorFlow, XGBoost, H2O and more.     Integrating SAS® Intelligent Decisioning and LLMs for Automated Analytical Decisioning How can you combine advanced analytics, Natural Language Processing (NLP) and decision orchestration to deliver exceptional customer experience and make analytically driven decisions at scale? Join this session to learn about the cutting-edge integration between SAS Intelligent Decisioning and large language models (LLMs), showcasing how AI and NLP can work together to drive real-time interactions.   Asset Monitoring Anomaly Detection at Scale Using Vibration Sensors Georgia-Pacific had a problem: unplanned equipment outages were costing the company money and disrupting operations. Then they started using vibration sensors for predictive maintenance to enable asset monitoring to harness the data they produced and transform it into actionable intelligence. Learn about how Georgia-Pacific implemented the end-to-end process with SAS® to analyze this vibration data and generate alerts on hundreds of assets with thousands of models in a distributed environment.     Navigating the Ethical Landscape of Generative AI Generative AI has the potential to revolutionize the way we write, develop, create, and solve problems. However, like any other system, it can introduce ethical risks if not deployed responsibly. Executives must understand the advantages and disadvantages of generative AI to guide their organization towards successful implementation. This session will equip the audience with key questions and considerations for conducting an ethical inquiry into generative AI from an executive perspective.   Enforcing Trustworthy AI Standards in Analytical Projects SAS Communities readers may be familiar with our release of the Trustworthy AI Life Cycle Workflow earlier this year. Now join yours truly as we how show SAS tooling makes NIST recommendations easier to adopt and how to ensure that analytical models don’t create disparate impact or harm. Attendees to this session will gain a greater understanding of trustworthy AI best practices as well as user roles and expectations for building responsible analytics. We'll also demonstrate how SAS supports a production model with documentation showing that an organization has done its due diligence for a fair model with processes that don’t cause undue harm.     LLMOps With SAS® Model Manager Can’t get enough of LLMs? Neither can we! Delve into the critical aspects of managing and operationalizing LLM models effectively. In this session, we will explore key topics such as prompt versioning, testing, output comparison and red teaming to ensure that your machine learning operations are efficient, secure and adhere to your company brand guidelines.   Be the Conductor With Process Orchestration for SAS® Viya® Need to orchestrate complex ModelOps process between multiple tools? Then we have the session for you! Using Apache Airflow as the flow execution engine, this new feature enables users to craft and manage custom flows across the apps in Viya. Offering robust access controls, dynamic partitioning of SAS tasks and nested sub-flows, process orchestration seamlessly integrates with the APIs in Viya to support complex business processes.   Bending the Rules: How to Use Linguistic Models to Improve the Performance of LLMs We have another great session on LLMs! Linguistic rules can be used to curate data and aid in training, fine-tuning, and prompt-tuning, as well as evaluating the results generated by LLMs. By combining linguistic rule-based models with LLMs, we can help to improve the quality and accuracy of LLMs and enable them to perform better on various tasks while cutting costs.     AI Governance: From Compliance to Strategic Differentiator Every disruption creates opportunities for organizations to emerge as new thought leaders. The impending release of the European Union AI Act spurred PZU, Poland’s largest insurance company, to prepare the enterprise for compliance and beyond. We'll tell you about PZU’s initial successes and lessons learned as it transitioned to modern, future-proof AI governance in collaboration with SAS’ Data Ethics Practice. Hear practical tips for strategies, how to make AI governance a strategic differentiator and how SAS resources can help with your governance journey.   LLM ModelOps in Action: A Guide to Using SAS® Model Manager for Large Language Models Last session on LLMs (I promise). In the rapidly evolving landscape of AI, large language models (LLMs) have revolutionized the way we interact with technology, offering unprecedented capabilities in generating humanlike text. However, the true potential of these models can only be unleashed when they are efficiently deployed and managed. This demo introduces participants to SAS Model Manager, a comprehensive ModelOps solution designed to streamline the management, deployment and monitoring of a wide range of machine learning models. Through a live demonstration, attendees will learn how to leverage SAS Model Manager for containerizing, deploying and serving LLMs, ensuring users can harness the full power of AI innovations in their own projects.   Visit the ModelOps Booth in the Innovation Hub The Innovation Hub will feature dozens of booths covering data management, machine learning, visualization, industry solutions, programming and more. The ModelOps booth will be staffed with experts from teams across SAS that are ready to answer your questions, hear your feedback, and provide product demos. The ModelOps booth will be in the SAS Spotlight area of the Innovation Hub, next to the Decisioning booth. The Innovation Hub is open for most of the conference, but check the Agenda at a glance for the most up-to-date hours.       Ready to Learn More? Whew, that is a lot of great sessions! Which one are you most excited for? Did I miss a must-see session? Let me know in the comments below!   Register for SAS Innovate now (if you haven’t already) to get the best deals! Want to learn more about what SAS Innovate has to offer? Then check out these resources: SAS Innovate Home Page 5 reasons to participate in the SAS Hackathon boot camps 7 sessions you don’t want to miss at SAS Innovate What’s happening at the SAS Innovation Hub? Unveiling the marvels of SAS Innovate: How it all comes together   ... View more

SAS has just released an experimental version of our Trustworthy AI Life Cycle Workflow for use with SAS® Model Manager and SAS® Workflow Manager on SAS Viya 2024.01 and later. Our Trustworthy AI Life Cycle workflow enforces standards and best practices set by the AI Risk Management Framework defined by the National Institute of Standards and Technology (NIST). The workflow allows organizations to document their considerations of AI systems’ impact on human lives.  Our workflow includes steps to ensure that the training data is representative of the population that is impacted, as well as that the model predictions and performance are similar across protected classes These steps help ensure that the model is not causing disparate impact or harm to a specific group. Furthermore, you can ensure that your model remains accurate over time by creating human-in-the-loop tasks to act when additional attention is needed.     The aim of the workflow is to make NIST’s recommendations easier to adopt for organizations by specifying individual roles and expectations, gathering required documentation, outlining factors for consideration, and leveraging automation to ease adoption. The result of the workflow is a production model with documentation to support the assertion that the organization has done its due diligence to provide evidence that the model is fair, and their processes do not cause harm.   Given the growing landscape of Responsible and Trustworthy AI, this workflow has been marked as experimental to gather your feedback. As you use the sample workflow, we would like to know what works well and what can be improved upon. Feedback can be provided in the Issues tab of our SAS Model Management Resources GitHub page.   To get started using this workflow, perform the following steps:   Perform first-time configuration steps for SAS Workflow Manager, if not already configured. Download the sample workflow definition from the SAS Model Management Resources GitHub page. Import and activate the workflow definition using SAS Workflow Manager. Workflow administrators should review timer and role defaults to ensure the values match their organizational processes. When a modeling project is available within SAS Model Manager, you can start the workflow. You will be prompted to specify the model owner. The workflow consists of user tasks, which will appear within the Task category in SAS Model Manager. Tasks only appear for the users specified in the task role to ensure that right individuals are providing documentation and approvals. The first task will be for the Model Owner in the Tasks category. Note: Users might need to refresh the tab to see new tasks. Several of the tasks involve providing documentation, aligning with NIST recommendations. To help compile this documentation, a companion template document in provided in the SAS Model Management Resources GitHub page, so be sure to download it and store it among the model files.   This workflow has several pieces to support the life cycle of a model. Let’s dive into each section.   Identify Stakeholders   Analytics is a team sport with data scientists, IT/engineering, risk analysts, data engineers, business representatives and other groups all coming together to build an AI system. Once the model owner is identified, they must then specify the other project roles. This workflow supports both single users and groups in the stakeholder roles. Additionally, users or groups can be specified for multiple roles. The roles used in the workflow are:   Model Owner – the decision maker for the analytics project. They will be responsible for documenting the purpose of the project and will provide approval or feedback as each piece of the project is assessed for Trustworthy AI practices. Model Developer – the data scientist(s) that develops the model. Model Engineer – the engineering or IT resource(s) that deploys the model. Model Risk Owner – the risk analyst(s) or risk manager(s) that manages and document the model risks. Data Engineer – the engineering resource(s) that prepares the data for modeling. Domain Expert – a resource that is available to address business or domain questions.   Verify Project   This section consists of mostly system tasks to confirm that the project metadata has been populated and that models exist within the project. If information or models are missing, users are promoted to correct the problem. Consider this section a completion check prior to advancing further.   Documents Project   The stakeholders listed previously are responsible for documenting various pieces of the project in alignment with NIST’s AI Risk Management Framework. In this section, responsible parties will be prompted to provide documentation via the tasks in SAS Model Manager. To aid in this compilation, we recommend using the provided template.   The documentation compiled in this step examines the proposed model usage, end users, expected performance of the AI system, strategies for resolving issues, potential negative impacts, deployment strategies, data limitations, potential privacy concerns, testing strategies, and more. Users can add additional documentation to cover any additional needs or use cases not covered by the workflow. By compiling this documentation, the various stakeholders can align on potential risks posed by the project as well as confirm the project plan.      Assess Data   The section for assessing data addresses two key factors: privacy and bias. Data privacy and bias are not concerns for all analytical projects. For example, some machine predictive maintenance projects rely on telemetry data and will not include information about individuals. For these cases, there is an option to by-pass the data privacy and bias questions.    For use-cases where Personally Identifiable Information (PII) is not required, the workflow prompts the data engineer to remove or mask PII. Otherwise, the risks of leveraging PII for modeling must be documented. Keeping PII data within the analytical base table increases risk of private information being leaked to bad actors.   Next, the workflow examines potential data biases, such as the inclusion of protected class variables or proxy variables for modeling as well as the representativeness of the data. Potential bias risks must be addressed or documented before moving forward. Using training data that is not representative of the target population can create a model that is less accurate for specific groups, resulting in undue harm for those groups. Including proxy variables or protected class variables in the training data can create a model that treats groups differently. For some use cases, such as health care, these variables may be important predictors for disease risk. For others, such as models aimed to predict who should receive a service or a benefit, this can lead to discrimination.     Assess Model   Models are assessed against three factors: performance, fairness, and explainability. Performance data provides information on how well the model can predict the desired event, which helps us understand how useful the model may be for our AI system. For gathering model performance metrics on training, testing, and validation sets, SAS provides functionality in support of several modeling types. For models developed in the graphical pipelines of SAS Model Studio, this information is calculated automatically and will be registered with the model. For models developed programmatically in SAS, we recommend providing the scored data to the macro at registration time. Similarly, this information can be calculated using the scored data using functions provided in the python-sasctl and r-sasctl open-source packages for python and R models.   For models that require an evaluation of potential bias, the model developer is prompted to calculate and provide the model’s performance and average prediction for each group of the protected class.  A model that performs worse on specific groups may point to problems with a lack of representation within the training data. Additionally, a model that is less accurate on specific groups may cause disparate impact or harm to those groups. Depending on the use case, a model that predicts different outcomes based on those groups may also cause undue harm. For models developed in SAS Model Studio, data scientists can select which variables to asses for bias in just a few clicks. Outside of SAS Model Studio, users can leverage the Assess Bias Action Set. Python-sasctl provides a function wrapping this action set for Python developers. When bias is detected in a model, model developers can leverage the Mitigate Bias Action Set to help mitigate model bias.   The last piece models are assessed against is model explainability. Model explainability helps stakeholders understand how the inputs to a model affect the model’s prediction. By understanding the relationship between the inputs and prediction, data scientists and domain experts can determine if the model captures well-known connections between factors. Additionally, explainability can provide evidence that a model is fair by demonstrating that a model is using factors that are unrelated to protected group status. Some models lend themselves to easy explainability. For example, the form of regressions and decision trees directly define how input variables are used to calculate the model prediction. Other models, such as gradient boosting or random forest, are hard to interpret and require additional methods to explain the model. There are several model interpretability techniques available including:   Local Interpretable Model-Agnostic Explanations (LIME) Partial Dependence (PD) Kernel Shapley Additive Explanations (Kernal SHAP) Individual Conditional Expectation (ICE) Hyperparameterized Shapley value estimation (HyperSHAP)   In SAS Model Studio, users can specify to run model interpretability techniques in the Post Training Properties of the Supervised Machine Learning nodes. For programming based models, users can explore the Explain Model Action Set.     Select Champion Model   The champion model is the best candidate for production. Users can mark a single model as the champion model but can mark multiple challenger models within the project in SAS Model Manager. This section of the workflow checks if a champion model exists, and if not, prompts the Model Owner to select a champion model before moving forward. When selecting a champion model, it is important to keep in mind performance and fairness assessment results from the previous steps. Additionally, the Model Owner should ensure the model qualifies based on qualities like the appropriateness of variables, explainability, and others. This model will undergoing further testing and approval in the next step before deployment to production.   Test and deploy model   Prior to production, the champion model should be tested using the framework specified in the project documentation. These tests may involve ensuring that the model can score data within SAS Viya and can score data within the publishing destination. Models should also be tested against unseen values and unseen data sets, such as a testing or validation set, to confirm performance continues to meet expectations.   After a model is tested and approved for production, it can be deployed for production. To prevent a model from going to production before approvals are attained, we recommend leveraging permissions within SAS Viya to prohibit user publishing. Additionally, administrators may need to do one-time configuration of their publishing destinations if they are not using an out-of-the-box destination. SAS Model Manager supports a variety of destinations, including a REST API endpoint hosted within SAS Viya, called MAS. SAS Model Manager also supports containerizing models, which also leverage a REST API endpoint, but will be hosted outside of SAS Viya. Other destinations include CAS and select databases.   Once the model is deployed, the model engineer will need to ensure that the model is incorporated in their business processes, whether that be using the REST API call within their web applications or creating a dashboard to review the results of batch scoring.     Monitor model and review project   All models decay, but models don’t decay at the same rate. Model decay leads to a decrease in model accuracy, so any decision being made using that model will be wrong more often. SAS Model Manager supports Performance Monitoring to pinpoint when decay begins to occur. Performance Monitoring can be combined with Key Performance Indicators (KPIs) to create thresholds for alerting. For this section, data scientists or engineers should ensure project performance monitoring and KPIs are defined. When model performance no longer meets the predefined threshold, the model owner must decide which step to take next, whether it be to retrain the model, select a new champion model, end the workflow, or retire the project.      Recurring Audit   The NIST AI Risk Management Framework recommends reviewing the project on a recurring basis to validate that the project parameters, assumptions, stakeholders, use case, data, and model are still valid. This provides a regular audit of AI Systems, allowing organizations to remove systems no longer in use. Retiring unused systems help save on cloud costs, reduces the number of models managed, and reduces the attack space for bad actors.     Once you have tried out and reviewed the workflow, we ask that you provide your feedback in the Issues tab of our SAS Model Management Resources GitHub page. Did this workflow fit well in your ModelOps processes and if not, what can we improve upon? Were the documentation and tests valuable to supporting your organization’s needs for Responsible AI? Are there factors that you would like to see considered for the next iteration? What additional can we provide to ease adoption? We will actively review provided feedback and incorporate what we can into the next iteration of this workflow!   Want to learn more about building Responsible AI applications? Check out the following resources:   A Comprehensive Approach to Trustworthy AI Governance Trustworthy AI Using SAS®: A Technical Discussion Trustworthy AI: 3 reasons we need it now How to Think About Trustworthy AI in 2024 and Beyond On-Demand Webinar Accountability in AI: A key ingredient for trust Inclusivity: A guiding principle for responsible innovation Data quality: The foundation for trustworthy AI The ethics of responsible innovation: Why transparency is key ... View more

SAS Model Manager supports both model and project versioning to help track changes and preserve model snapshots across important milestones. Versioning is a key capability that supports proper model governance and validation processes. For today’s article, let’s review how to use project and model versioning as well as discuss recent enhancements to versioning.   Project Versioning    Projects hold multiple models and the information specific to one analytical use case. Projects can be versioned within SAS Model Manager, but project versioning has a few key differences from model versioning. For example, each project version holds a distinct set of models. Commonly, a new project version is created whenever the training data for the set of models updates, such as during model retraining. In fact, whenever a user opts to retrain their modeling projects from SAS Model Manager, a new project version is generated automatically.   Project versions can be created, deleted, renamed, and viewed from the Version drop-down in the Models tab of the project. This short video demonstrates how to interact with project versions:     Model Versioning    Each version of a model holds a variety of files and information, including the score code, model ASTORE or pickle file, properties, link to the training table, requirements, and more.   From the Versions tab of the model, users can view each version of the model as well as create a new model version. This short video demonstrates how to interact with model versions:   As of writing just after the release of SAS Viya 2024.01, a new model version is generated by the following activities: Publishing a model. Setting a model as champion. Making changes to the original model within SAS Model Studio and re-registering the model. Manually creating a new model version.   Whenever a new model version is created, the previous model versions are locked from editing. Locking the previous model versions from editing ensures that the files and information we have within SAS Model Manager accurately reflects what is available in deployment destinations.   Users have provided several pieces of feedback for how we can improve versioning to better fit their needs. We have already released several new capabilities based upon that feedback and are currently working on addressing more of that feedback.  So next, let’s review what’s new and on the horizon for versioning within SAS Model Manager.   What’s New with Versioning in SAS Model Manager   Prior to our recent changes, users were restricted to only use the latest version of a model for decision flows, score testing, publishing, and performance monitoring. After discussions with our users, we decided to remove that restriction. Allowing users to leverage previous model versions provides greater flexibility in model validation and governance processes. Let’s dive into how you can start using each of these new capabilities.   Use Any Model Version within a Decision Flow   Since SAS Viya 2023.05, SAS Intelligent Decisioning users have been able to specify which version of a model is used within a decision flow. After a model is added to the flow, users can select the model version from the properties pane:     This allows users to create decision flows using a specific model version or opt to always pick up the latest version of a model. Multiple versions of a model can exist within the same decision flow, enabling A/B Testing of new model versions against previous ones.   Score Test All Versions of a Model   With the release of SAS Viya 2023.09, any version of a model can be selected for a score test. After users select their model inside a scoring test, they can then select which version of that model they would like to use.  Users have two options for selecting a model version to use for score testing: Select a model version by number: this option will create a scoring test that will always use the selected version of the model. Select to use latest model version: this option will always use the latest version of a model. If a new version of a model becomes available, re-running this test will use the newest version of a model. These two options give users a static option and a dynamic option for creating scoring tests.   Creating scoring tests for previous model versions allows users to continually test their production and pre-production models and provide stronger model validation over time.   To learn more about this new capability, see the following demo:     Publish All Versions of a Model   From SAS Viya 2023.10 onwards, any version of a model can be published. From the publishing dialogue, users can select which version of a model they would like to publish. This means that the same version of a model can be deployed in multiple destinations, such as a testing environment and a production environment. This also allows for easy roll-back of a production model to a previously deployed version.   To learn more about this new capability, see the following demo:     Monitor All Versions of a Model   Beginning on SAS Viya 2024.01, any version of a model can be monitored. Like score testing, users have two options for selecting a model version to monitor: Select a model version by number: this option will always monitor the selected version of the model. Select to use latest model version: this option will always monitor the latest version of a model. If a new version of a model becomes available, re-running the performance monitoring report will use the newest version of a model.   Monitoring the performance of a model across versions can help users react to decay if multiple versions of a model are deployed across destinations or determine if a production model should be rolled-back to a previous version.   To learn more about this new capability, see the following demo:     Upcoming Version Enhancements   Our users have reported that they would like automatic version incrementing (also know as version bumping) when publishing a model or marking a model as champion to be optional. This is the next piece of feedback we would like to address for versioning enhancements. It is necessary to continue to lock models from editing when publishing or marking a model as champion to ensure models within SAS Model Manager match what is available in external destinations and environments, but we can let the user decide when to create a new model version.   Versioning is a key component for robust model governance and management. Based on user feedback, we have made adjustments to better support their processes and use cases.  Interested in providing your own feedback to the SAS Model Manager team? SAS Product Suggestions is a board for product feedback and ideas.     Want to learn more about SAS Model Manager? Then check out the following resources: A Tour of SAS Model Manager Ask the Expert - How Do I Use SAS Model Manager?  Ask the Expert - Mastering ModelOps: Supercharge Your Analytics Journey  IDC MarketScape: Worldwide Machine Learning Operations Platforms 2022 Vendor Assessment ModelOps Explained: A Starter’s Guide to Deploying and Managing AI and Analytical Models ... View more

As promised last year, this month we released a new step to register Python models from SAS Studio Flows directly into SAS Model Manager. Flows in SAS Studio represent a sequence of operations, where each step is a node on the flow canvas. Data and information flow between the steps. Since flows are commonly used to prepare data and train models, a natural last step of the flow is to register the trained model into SAS Model Manager. In SAS Model Manager the model can be compared, managed, tested, validated, deployed, and monitored alongside all models developed for an analytical use case. Last year, we released a step to register SAS Models and with 2024.01, we now have a step for Python models too!   To get started using this new step, do the following: Add a Table node to the flow canvas. Fill out and run this node to pull your training data into the flow. If your data resides in CAS, be sure to start a CAS session first! Add a Python Code node to the flow canvas. Link the Table node to the input port of the Python Code node. Next, open your Python code node. There will be tabs for the Python code, Node options, and Notes. Navigate to the Node tab. From the Node tab, you can see the macro variable used to represent the Table flowing into the Python Code node via the Input Port. Now in your python code, you can create a python data frame from the input data using:  dfin = SAS.sd2df(_input1)​ Using the Python Code node, write your model training code. At the end of your training code, use your trained model to score your dataset.  From the Node tab, you can also find the macro variable used for data flowing out of the node via the Output Port. Save your scoring data using:   SAS.df2sd(scored_df, _output1)​ Add another Table node to the canvas and link the output port of the Python code node to this Table. Specify where the table should be saved as well as a name for the table. Run the Python Code node and the Table node to populate the table with the scored data. Add the Register Python Model step to the canvas and link the scored data to the input port of the step. Fill out the step with a name for the model, the trained model name used in the training code, the SAS Model Manager project name, the target variable, the target values, the target event value, the input values, the output classification variable, and the output event probability variable. Run the node. Open the project in SAS Model Manager and find an executable version of your Python model, ready for governance, management, deployment, and more!   Optionally, you can reference a Python code file that is not a part of the flow, but it will need to be saved to the file system of SAS Viya and have been run within the same SAS Studio session.   To see the step in action, check out this demo video:   What step would you like us to develop next for easy integration between SAS Model Manager and SAS Studio? Add your suggestions to SAS Production Suggestions to let us know!   Want to learn more about SAS Studio flows? Check out the following resources: Go with the Flow: New SAS Studio Step for Model Registration Build Flows with SAS Studio Video Tutorial SAS Studio Flows on Viya: Prep data, build ML model, score data and register model - Video Tutorial Building SAS Studio Flows in SAS Viya Course SAS ®  Studio Flows and Steps - Free Ask-the-Expert Webinar ... View more

October was a BIG month for new Python functionality in Python-sasctl and SAS Model Manager. SAS Model Manager is SAS’s leading solution for helping organizations manage, test, audit, deploy, monitor,  and search their models. And yes, this includes Python models. Data scientists using Python lean on the Python-sasctl package to interact with SAS Model Manager and ensure that their models are included within a wider ModelOps and MLOps strategy. And we’ve release three new capabilities in the v1.10.0 release of python-sasctl, including a new function to assess model bias, Key Performance Indicator (KPI) and Hyperparameter integration, and support for automatic score code and metadata generation for TensorFlow Keras models. Let’s dive into each.   Assess Bias for Python Models   Python-sasctl now provides a function to assess the bias and fairness of Python models! This function leverages SAS’s Assess Bias action to take pre-scored data and determine if there are differences in the model’s accuracy and predictions across values of the specified sensitive variable, such as race or gender.  The new function within python-sasctl can return the following results: Group metrics, which contain the model assessment statistics and average prediction for each level of the sensitive variable. Max differences, which contain the maximum absolute difference between all pairs of levels from the sensitive variable. Bias metrics, which contain the bias metrics for the predictive model. The results of these tests can help support that a model is treating groups similarly or let the data scientist know that they need to take action to build a fair model.   The new assess model bias function can be called using the following parameters:   pzmm.JSONFiles.assess_model_bias( score_table, # table scored by the model actual_values, # variable with the actual target valables sensitive_values=, # variable to assess for bias, can add more as a list prob_values, # variables for predicted probabilities (classification)s levels, # same order as prob_values json_path, # directory for json files, optional return_dataframes=True # returns group metrics, bias metrics, and max differences )   To learn more, see this example and the following demo:     Key Performance Indicator (KPI) and Hyperparameter Integration   SAS Model Manager generates Key Performance Indicators (KPIs) automatically based on performance monitoring definitions. However, we can extend the KPIs tracked by SAS Model Manager through the addition of custom KPIs.  From Python, we can perform local tests on models we’ve registered to SAS Model Manager, then pass up the resulting values as custom KPI values. For certain python models, sasctl will also generate a json file containing the hyperparameters of the model, making them easily accessible for future use.   To learn more, see this example and the following demo:     TensorFlow Keras   Keras is the high-level API of the TensorFlow platform. It provides an approachable, highly productive interface for solving machine learning (ML) problems, with a focus on modern deep learning. And now, python-sasctl can generate the metadata files and score code to run models developed using TensorFlow Keras in SAS Model Manager! To learn more, see this example and the following demo:     What would you like to see next for python-sasctl? Let us know in the comments!   To learn more about Python-sasctl and SAS Model Manager, check out the following resources: MLOps Uncoiled: Python’s Path on SAS Viya with SAS Model Manager Python-sasctl GitHub Score Millions of Records in Minutes Using Python Models: Python Container Scoring Optimization MLOps for Pirates and Snakes: The Sasctl Packages for R and Python IDC MarketScape: Worldwide Machine Learning Operations Platforms 2022 Vendor Assessment ModelOps Explained: A Starter’s Guide to Deploying and Managing AI and Analytical Models SAS Viya & Open Source ... View more

Python-sasctl is an open-source package provided by SAS to ease the handoff between the data scientists developing the model and the MLOps engineers managing and deploying the model. Through Python-sasctl, data scientists can take their models developed using Python packages, such as sklearn or xgboost, and automatically generate the scoring code, model pickle file, input variables metadata, output variables metadata, package and version requirements, training performance metadata, and model properties in just a few lines of code. Next, these files are directly pushed into SAS Model Manager.     SAS Model Manager offers a unified model repository where models can be organized, audited, tested, monitored, and deployed. Python models can be deployed into a variety of locations, such as a real-time REST API endpoint on SAS Viya, into a batch destination on SAS Viya, or into a container. A container packages the model and its dependencies. This allows it to run on-prem or in any cloud.     Since SAS Explore took place in Las Vegas this year (and I didn’t want to miss any of the fun), for my session MLOps Uncoiled,  I trained several models in Python to predict whether my flight would be delayed 15 minutes or more. Next, I leveraged Python-sasctl to generate the metadata and executable scoring model.  Python-sasctl supports automatic score code and metadata generation for many popular Python frameworks, including Sklearn, Xgboost, MLflow, TensorFlow Keras, H2O, and more. Finally, I used Python-sasctl to register the model into SAS Model Manager, where it can be compared, tested, monitored, and deployed.   Python-sasctl helps generate several pieces of metadata. To explain why we collect each piece, let’s dive into the anatomy of a model.     To ensure I collect all the rich metadata MLOPs Engineers want, I follow a specific pattern when registering Python models: Initialize variables specific to our model. This make reusing the same block much easier! Create a sub-folder to hold model files. This keeps everything organized. Save a binary representation of the model. For most Python models, this is the pickle file. Create metadata files. The more we provide to our ML Engineer the happier they will be! Since I strive to act in best practice, I will generate them all. These files include: The inputs my model expects The output my model generate My model's performance on the training and testing data Basic model information, like modeler, algorithm, target variable, etc. Package and version requirements Next, I use the import model function to write our score code, zip the model files together, and deliver the zip folder directly into Model Manager, where our ML Engineer can start managing my models.   The following demo is a clip from my SAS Explore session highlighting data visualization, model training, model registration, and model management.     To learn model about Python-sasctl and SAS Model Manager, check out the following resources: Python-sasctl GitHub Score Millions of Records in Minutes Using Python Models: Python Container Scoring Optimization MLOps for Pirates and Snakes: The Sasctl Packages for R and Python IDC MarketScape: Worldwide Machine Learning Operations Platforms 2022 Vendor Assessment ModelOps Explained: A Starter’s Guide to Deploying and Managing AI and Analytical Models SAS Viya & Open Source ... View more

This session introduced R-sasctl and demonstrated how to leverage the package to bridge the gap between data scientists and MLOps engineers for R models.   R-sasctl was released in January of this year and  in our SAS Model Manager community. R-sasctl helps generate the score code and metadata for R models as well as allows R developers to import their model into SAS Model Manager directly from their R development environment. This makes the hand-off between our data scientist developing in R and our MLOps engineer managing and operationalizing the model much easier. R-sasctl has several other neat functions as well, but we are going to focus on the registration today.   To learn more about R-sasctl, check out the following resources: MLOps for Pirates: R-sasctl  R-sasctl’s documentation R-sasctl’s GitHub Presentation slides are attached to this post. ... View more

R-sasctl was released in January of this year and  in our SAS Model Manager community. R-sasctl helps generate the score code and metadata for R models as well as allows R developers to import their model into SAS Model Manager directly from their R development environment. This makes the hand-off between our data scientist developing in R and our MLOps engineer managing and operationalizing the model much easier. R-sasctl has several other neat functions as well, but we are going to focus on the registration today.     R-sasctl helps generate several pieces of metadata. To explain why we collect each piece let’s dive into the anatomy of a model.     First, the score code and scoring resources are necessary for scoring new data, both within SAS Model Manager, like score testing or performance monitoring, or outside of it, like publishing into CAS or containers.  Next, the input and output variables are critical for knowing what data the model expects as well as what outputs it generates. It also gives our engineers and project owners a quick view of what variables the model uses, to ensure only approved data sources were used for modeling. Following, properties tell us which file represents which role as well as who our modeler was, what algorithm they used, and more. These are all important factors when it comes to auditing your model. And finally, our diagnostic files include numerous fit statistics, lift charts, roc curves and more. This helps us compare our models on accuracy to know which model is the best fit for our use case.   The following demo video will take you through the training and registration process for a R model within R Studio. At the end of the video, we will get a view of the model within SAS Model Manager, where it is managed, tested, monitored and deployed.   Demonstrated above are just a few of R-sasctl’s capabilities, but there are several things we didn’t touch on, including R-sasctl’s support for PMML models, the project management capabilities, model publishing, and execution of published models, including parallel execution of batch data. Check out R-sasctl’s documentation to learn more about these other neat functions! What to try out an example yourself? See R-sasctl’s GitHub page.   Have you tried R-sasctl out? Tell us your thoughts below! ... View more

SAS Viya is fast. Earlier this year, an independent group analyzed and reviewed the speed at which SAS Viya could load and pre-process data, train and score models, and calculate accuracy. They found that on average, SAS Viya was 30 times faster than all other solutions in the study. And it makes sense that SAS Viya is so fast. SAS has been in the analytics game for almost 50 years staffed by the best statistical programmers and computer scientists who can optimize not only the engine running the code but also the code itself.   In my tests, I have seen SAS Viya score 10 million records of data using a SAS model in about 2 minutes. But what about a real challenge? How fast can SAS Viya run non-SAS code in a non-SAS engine? The SAS Model Manager team was challenged to run Python models faster. At minimum, the team was tasked with scoring 10 million records using a Python model in 45 minutes. I am happy to report that the SAS Model Manager team not only met that challenge but exceeded it! After optimizing our Python base container, SAS Model Manager can deploy Python models into a container in 2-3 minutes. Once the container is built and running, the team’s benchmarks show that a single container can score 10 million records of data in under 3 minutes. That is over 15 times faster than our goal. Additionally, that means we are scoring 60,000 rows per second!     SAS Model Manager is the ModelOps solution for registering, managing, comparing, testing, monitoring, and deploying models on SAS Viya. SAS Model Manager can build and deploy containers for SAS, Python, and R models using a technology called SAS Container Runtime. SAS Container Runtime allows models to run in any OCI-compliant system using a lightweight and scalable container that manages the dependencies required by that model. These containers are executed outside of SAS Viya without needed to pay any additional fees to SAS.      To leverage the faster scoring of Python models inside a container, you will need SAS Viya 2023.09 or later. Next, you will need to: Configure the container registry where SAS Viya will push the container. Registries in Docker, Azure, GCP, and AWS are supported. Configure the container publishing destination in SAS Model Manager. Register the Python model into SAS Model Manager with the requirements.json file. This file tells SAS Model Manager what packages and package versions the Python model needs to run so that the packages are included in the container. If you need help getting the model into SAS Model Manager with the requirements file, don't fear. Python-sasctl can help. From SAS Model Manager publish the Python model to the container publishing destination.  Run the container. If you are leveraging Docker, you can the following command to start your container on the specified port:   docker run -p <port-number>:8080 <container-name>    Send data to the container via REST API. If you have published Python models to a container prior to 2023.09, it is important to note that new endpoints are available that standardize with the SAS-base container, which had a streamlined REST call. The old endpoints still work as expected, but will be slower. If you are calling your container via Python, you can use the following block of code:   # Specifiy URL url='http://localhost:<port-number>/<model-name>' # Make POST request with data response = requests.post( url=url, headers={"Content-Type": "application/json"}, data=json.dumps(data.to_json()) ) # View results new_container_results = pd.DataFrame(response.json()["data"]) pprint.pprint(response.json()["metadata"]) pprint.pprint(new_container_results)   The following demo shows the container in action:     Want to learn more? Check out these resources! IDC MarketScape: Worldwide Machine Learning Operations Platforms 2022 Python-sasctl Examples How Do I Use SAS Model Manager? Ask the Expert Webinar SAS Viya & Open Source What's New with SAS Model Manager? SAS Runtime Container ModelOps and DecisionOps for the Modern World ... View more

Data scientists build models to improve outcomes through better decision making. For their models to be effective, their models need to integrate with a business decisioning process. If our outcome is to reduce fraud losses, our model needs to be integrated into the real-time payments process to approve transactions as they appear. But if our outcome is to increase clicks from a personalized weekly email blast, our model needs to run once a week against all email recipients. To accommodate varying analytics needs, SAS Model Manager can deploy models into both batch and real-time destinations. In my previous article, we discussed how to score data over REST API using models deployed into the SAS Micro Analytics Service (MAS). In this article, we will focus on using SAS Cloud Analytic Services (CAS) for batch.    SAS Cloud Analytic Services provides the run-time environment for data management and analytics on SAS Viya. Thus, you will hear CAS referenced as a place to store data, but also as an analytical engine. CAS operates in-memory to provide the best performance for big data.   CAS requires that users communicate via a session. To leverage CAS programmatically, users must first start a session. Sessions are used for authentication, fault isolation, efficient scoping of resources, and resource tracking.   Deploying and scoring a model using CAS takes just a few steps. In fact, if you are using SAS Model Manager, you can score data using CAS via a Scoring Test in just a few clicks. From the Scoring tab of a Model Manger project, click New Test. Next, give your test a name and select the model, input table, and output library before clicking Run.     When your test completes, a green check mark will appear under the Status column. Next, click the table icon under the Results column and then you can explore the results metadata, output data, code, and log.     You can also publish and run a model programmatically using PROC CAS or SWAT. Regardless, some teams prefer to leverage the APIs to build an automated process. In this article, I will walk through the steps using the APIs for scoring.   Step 1: Create the CAS destination   Unlike the MAS destination, the CAS destination is not available out-of-the-box. But that is not a problem! An admin can configure the destination in just a few steps from the SAS Viya CLI or from Environment Manager.   From Environment Manager in SAS Viya, first click Publishing Destinations. Then click the folder icon in the upper-right corner to create a new destination. Select the type as CAS, give the destination a name, select the CAS server, select the CAS library, and finally give the model table a name before clicking save.       Step 2: Publish the model to CAS   Now that we have configured our destination, we are ready to publish our model. We can deploy a SAS, Python, or R model into CAS. First, select your model(s) and hit the Publish button. From the Publish Models dialogue, select the CAS destination. Give your published model a memorable name before clicking the Publish button. In just a few seconds, we are ready to start scoring data over REST API using our model!     Step 3: Prepare to call REST API   REST APIs are a standard for communication between applications. Given the popularity of REST APIs, users have a variety of ways to call REST APIs. REST APIs can be called using tools like Postman or Insomnia, via cURL, using the requests package in Python, and many other methods. This allows users to call REST APIS using the method that is easiest for their skill sets or best fits their process needs.   I personally prefer programming in Python, but the following examples can easily be adapted to other methods. But just a note for our Python and R developers: instead of calling the APIs directly, you can also call CAS actions using the SWAT package. If you would like to see an article about scoring using models deployed to CAS over SWAT, please let me know in the comments! Otherwise, to call APIs in Python, I need to import two packages: requests and json.   import requests import json   Regardless of the method you use to call the REST APIs, you will need three things: the URL of your SAS Viya environment, your username, and your password. The following block prompts a user to enter this information in Python:   #Package to hide passwords import getpass host = input("Viya URL: ") username = input("Username: ") password = getpass.getpass("Password: ")   Step 4: Get Access Token   An access token allows us to make the REST API calls to SAS Viya. We can reuse our access token, but it will expire after a set time passes. To use your token longer, you can refresh your token or increase the timeout before the token expires.   This is a POST request to the /SASLogon/oauth/token endpoints with your username and password in the body of the request.   url = f"{host}/SASLogon/oauth/token" authBody = 'grant_type=password&username=%s&password=%s' %(username, password) headersAuth={'Accept': 'application/json', 'Content-Type': 'application/x-www-form-urlencoded'} r = requests.request('POST', url, data= authBody, headers=headersAuth, auth=('sas.ec', '')) token = r.json()['access_token']   Alternatively, there are a few other approaches you can take to creating your access token, as highlighted in developer advocate Joe Furbee’s blog and Simon Edward’s article.   Step 5: Start Session   To interact with CAS, we need to start a session. We can start a session by making a POST request to the /cas/sessions endpoint with our authorization token within the headers. We will need to save the session ID to make our future requests.   headers = {'Authorization': 'Bearer ' + token} url = f"{host}cas-shared-default-http/cas/sessions" r = requests.request('POST', url, headers = headers) session = r.json()['session']   Step 6: Load Data   Unlike MAS scoring where we pass a row of data, CAS scoring operates on a table in CAS. There are several methods for loading data into CAS. Using the Data Explorer on SAS Viya, under Manage Data in the main menu, we can build connections to a database or import a local file in just a few steps. We can also load data programmatically. For example, Peter Styliadis’s blog details one way to load csv files into CAS from Python.     Step 7: Make Scoring Rest Call   Using the method outlined by Joe Furbee in this blog, we can translate CASL code into REST APIs. I leveraged this method for the runModelLocal action.   Before we can score our data, we need to specify the input data, the model name and location, and where we want the data to end up. The following block prompts a user to enter this information in Python:   inlib = input("Input CAS Library: ") intable = input("Input CAS Table: ") modelname = input("Model Name: ") inmodel = input("Model CAS Library: ") modeltable = input("Model CAS Table: ") outlib = input("Output CAS Library: ") outtable = input("Output CAS Table: ")   Now we have everything we need to score the data using the model published into CAS. Our payload will include our data and model information, which will we pass as a POST request to the /cas/sessions/sessionID/actions/modelPublishing.runModelLocal endpoint, where we replace sessionID with the session ID we generated earlier.   headers = {'Authorization': 'Bearer ' + token, 'Content-Type': 'application/json'} payload = {"inTable":{"caslib":inlib, "name":intable}, "modelName":modelname, "modelTable":{"caslib":inmodel,"name":modeltable}, "outTable":{"caslib":outlib,"name":outtable}} url =f"{host}cas-shared-default-http/cas/sessions/{session}/actions/modelPublishing.runModelLocal" r = requests.request('POST', url, json = payload, headers = headers) r.json()   Step 8: Promote Data or Pull Data   Our table was scored within our CAS session and as such, is not available outside our session. But, with one more call we can promote our table so it is available globally, even after our session is gone. To promote our table, we will make a POST request to the /cas/sessions/sessionID/actions/table.promote endpoint. We will also need a payload compiling the name and library of our table as well as a target location for our table after promotion.   headers = {'Authorization': 'Bearer ' + token, 'Content-Type': 'application/json'} payload = {"caslib":outlib, "name":outtable, "target":outtable, "targetLib":outlib} url =f"{host}cas-shared-default-http/cas/sessions/{session}/actions/table.promote" r = requests.request('POST', url, json = payload, headers = headers) r.json()   Conversely, we may not want our table promoted globally. We may just want to view our table now. We can pull data from CAS using the fetch action. To fetch a table from CAS, we will make a POST request to the /cas/sessions/sessionID/actions/table.fetch endpoint. In our payload, we will specify which table we want to pull as well as how many rows we want to pull.   headers = {'Authorization': 'Bearer ' + token, 'Content-Type': 'application/json'} payload = {"maxRows":10, "table":{"caslib":outlib,"name":outtable}} url =f"{host}cas-shared-default-http/cas/sessions/{session}/actions/table.fetch" r = requests.request('POST', url, json = payload, headers = headers)   If we are pulling this data via Python, we can save it as a data frame using pandas.   import pandas as pd columns = [] for x in (r.json()['results']['Fetch']['schema']): columns.append(x['name']) df = pd.DataFrame(r.json()['results']['Fetch']['rows'], columns = columns) df   Step 9: Close session   Our session will expire after a set amount of time passes, but we can be good stewards of our resources by closing our session once we are done. We can close our session by making a DELETE request to the /cas/sessions/sessionID endpoint, where we replace sessionID with the session ID we generated earlier.   headers = {'Authorization': 'Bearer ' + token} url = f"{host}cas-shared-default-http/cas/sessions/{session}" r = requests.request('DELETE', url, headers = headers) r   A 200 response indicates that our session was closed successfully.   And now you are ready to score using models deployed into CAS over REST API! What other publishing destinations do you want to learn more about? Let me know in the comments below! Do any CAS power-users have tips and best practices to share? Add them below as well! ... View more

SAS Model Manager supports a stress-free handoff between the data scientist training the model and the MLOps engineer managing and the deploying the model. To easily pass models from the development environment to our unified repository, SAS Model Manager provides various methods for model registration. These methods were designed to register models using a similar approach to how the models were trained. Models developed using point-and-click in Model Studio, can be registered in a button click. Models developed using Python and R can be registered using the Python-sasctl package and the R-sasctl package. Models developed in SAS Studio can be registered using a SAS macro. Now with the rising popularity of Flows in SAS Studio, we want to provide another mechanism for registration.    We’ve been working with the SAS Studio team to create a new step to register SAS models. This step fits seamlessly within a flow. Data scientists can train their models and register them in the same flow! This step supports models that generate a Data Step score code, such as regressions and decision trees, as well as models that generate an Analytical Store (Astore) table.   To get started with this new step, do the following: In your model training code, be sure to save the model scoring resource. This will be the Data Step file or the Astore table. As a best practice, I also recommend saving the scored data table. This data table is used to generate the metadata required to compare models. For Astore models, you will need to add a Table node that references the Astore table. A Table node is also necessary for the scored data. Be sure to run all Table nodes before filling out the Register SAS Model Step. Add the Register SAS Model Step and attach the Table node(s) as input(s). If you need to add a second input port, right-click the register step and add an additional port from the options. Fill out the required information. To add the scored data, check the first box on the Data tab and fill out the information about the scored data table. Run the step and see the models within SAS Model Manager, ready to manage, deploy, and monitor.   To see the step in action, check out this demo video:     Given the support of Python code within SAS Studio flows, we are currently working on adding another step to Register Python models. Stay tuned for more! In the meanwhile, what enhancements would you like to see for the Register SAS Model step and what steps would you like to see next for SAS Model Manager?   Want to learn more about SAS Studio flows? Check out the following resources: Build Flows with SAS Studio Video Tutorial SAS Studio Flows on Viya: Prep data, build ML model, score data and register model - Video Tutorial Building SAS Studio Flows in SAS Viya Course SAS ®  Studio Flows and Steps - Free Ask-the-Expert Webinar   ... View more