This guide will walk you through adding Python code files to a decision flow in SAS Decision Builder, allowing you to customize approval and denial messages based on the decision outcome. We’ll also show you how to adjust a rule set on the fly to include necessary variables. Let’s get started! Step 1: Create Python Code File for Approval Messages   Create a New Python Code File: Start by creating a Python code file to generate an approval message. Write the Code: Enter the Python code that will send a custom approval message based on the decision outcome. Save the Code: Click Save. Upon saving, the system will automatically add any variables referenced in the code to the Variables tab of the Python code file.     Continue here to read the full article     ... View more

This guide will walk you through adding Python code files to a decision flow in SAS Decision Builder, allowing you to customize approval and denial messages based on the decision outcome. We’ll also show you how to adjust a rule set on the fly to include necessary variables. Let’s get started!   Want to see these steps in action? You can also watch our Integrating Python Code in Decision Flows video guide to add python code files to your decision flow in SAS Decision Builder. The video walks through each stage of the process, making it even easier to follow along and get started! Step 1: Create Python Code File for Approval Messages   Create a New Python Code File: Start by creating a Python code file to generate an approval message. Write the Code: Enter the Python code that will send a custom approval message based on the decision outcome. Save the Code: Click Save. Upon saving, the system will automatically add any variables referenced in the code to the Variables tab of the Python code file.     Step 2: Update Data Types for Variables   Go to the Variables Tab: In the Python code file, navigate to the Variables tab. Adjust Data Types for Specific Variables: Set LOAN to Decimal. Set TERM to Integer. Set OFFER_RATE to Decimal. This ensures that the data types match the data being processed.     Step 3: Create Python Code File for Denial Messages   Create Another Python Code File: Repeat the process to create a Python code file that generates a denial message. Write and Save the Code: Enter the code to send a denial message and save the file. Variables will again be added automatically to the Variables tab if referenced in the code. Step 4: Add Branches to Handle Different Loan Outcomes   Navigate to the Decision Flow: Go to the Loan_Request_Review decision flow in SAS Decision Builder. Add Branches for Different Outcomes: Right-click on the Ruleset step and select ‘Add Branch’. Create branches for the ‘Approved’, ‘Denied’, and ‘Review’ paths based on the Loan Status variable. This allows you to control the decision flow based on each loan outcome.     Step 5: Add Python Code Files to Each Branch   Add the Approval Message File: Right-click on the ‘Approved’ branch and choose Python code file. Select the Loan_Approved_Message file to send a customized approval message. Add the Denial Message File: Right-click on the ‘Denied’ branch and choose Python code file. Select the Loan_Denied_Message file to send a denial message.   Step 6: Ensure Required Variables are Available in the Decision Flow   Before running the decision flow, ensure that all necessary variables are set up, particularly the TERM variable that is used in the approved message python code file. Go to the Variables Tab in the Decision Flow: Check the Variables tab to confirm all required variables are present. Navigate to the Initial_Loan_Review Rule Set: In the decision flow, click on the rule set name (e.g., Initial_Loan_Review) to edit the rule set. Add and Define the TERM Variable: In the rule set, go to the Variables tab and add a new variable named TERM. Set the Data Type to Integer, mark it as an Output Variable, and set a fixed value of 24 (for a 24-month loan term). Save the changes and return to the decision flow.     Step 7: Run the Decision Flow and View Results   Run the Decision Flow: Click the Run button. Select the input dataset that contains loan application data. Choose a destination in your lakehouse for the output results. View Decision Outcomes: After running the decision flow, go to the destination table in the lakehouse to review the decision outcomes. Check the output messages to see the custom approval or denial messages generated by the Python code files based on the loan status.     And that’s it! You’ve successfully added Python code files to a decision flow, adjusted rule sets on the fly to include key variables like the loan term, and customized messages based on loan status outcomes. This process provides a flexible way to deliver personalized messaging within automated decision flows in SAS Decision Builder. ... View more

Welcome to SAS Decision Builder! This guide will walk you through creating a basic decision flow to automate a business process. Follow these steps to set up your first decision flow, using rule sets, variables, and validation to automate key decisions.           Continue here to read the full article ... View more

Welcome to SAS Decision Builder! This guide will walk you through creating a basic decision flow to automate a business process. Follow these steps to set up your first decision flow, using rule sets, variables, and validation to automate key decisions.   Want to see these steps in action? You can also watch our Get started with Decisions video guide to creating your first decision flow in SAS Decision Builder. The video walks through each stage of the process, making it even easier to follow along and get started! Step 1: Create a New Decision Flow   Click on ‘Create New Decision’ in SAS Decision Builder. Name your decision flow according to your business need (e.g., ‘Decision_Flow_Name’). Open the flow editor to start building the decision flow.     Step 2: Add a Rule Set to the Flow   Inside the flow editor, right-click on the Start decision step. Select ‘Add’ and choose ‘Rule Set’ from the menu. A dialog box will appear with a list of existing rule sets. Select a rule set that matches your decision criteria. This rule set will contain the logic to evaluate input data against predefined conditions. Lean more about How to Create a Rule set.     Step 3: Create and Define Variables   In the flow editor, go to the Variables tab to add the necessary input and output variables. Define the variables that your decision flow will use to process data, such as numerical values, categories, or other criteria specific to your business context. For each variable, mark it as an Input (if it’s coming from the input dataset) or Output (if it’s part of the decision outcome) by checking the corresponding checkbox.     Step 4: Review the Rule Set Logic   After adding the rule set, right-click on it and select ‘Open Rule Set’ to view the specific rules and conditions within it. Review the logic to ensure it aligns with your decision-making criteria, such as specific thresholds, conditions, or actions. Once you’re familiar with the rule set's logic, return to the decision flow editor to continue building the rest of the flow.     Step 5: Run the Decision Flow   Once the decision flow is set up, click on the Run button to execute the flow. A dialog will prompt you to choose an input dataset from your Microsoft Fabric lakehouse. Select the dataset that contains the data relevant to your decision flow. Next, specify the lakehouse destination for the decision outcome table, where the results will be saved. Click Confirm to run the decision flow.     Step 6: View the Decision Outcomes   After running the decision flow, go to the lakehouse destination you selected. Open the decision outcome table to view the results, which will show the decisions made based on your criteria (e.g., Approved, Denied, Flagged for Review). This table provides an immediate overview of how the flow processed the input data and generated outcomes based on the criteria defined in the rule set.     Congratulations! You’ve just created a decision flow in SAS Decision Builder to automate a business decision process. This flow enables efficient, data-driven decisions and provides real-time insights into outcomes.   Explore further to add advanced logic, Python code files to your flows for even more powerful decision-making. ... View more

Creating rule sets is an essential part of automating business decisions in SAS Decision Builder. This guide will walk you through setting up a rule set, defining variables, adding rules, incorporating the rule set into a decision flow, and viewing the results. Let’s get started!           Continue here to read the full article ... View more

Creating rule sets is an essential part of automating business decisions in SAS Decision Builder. This guide will walk you through setting up a rule set, defining variables, adding rules, incorporating the rule set into a decision flow, and viewing the results. Let’s get started!   Want to see these steps in action? You can also watch our video guide to creating a rule set and using it in a decision flow in SAS Decision Builder. The video walks you through each stage, making it easy to follow along and get started. Watch the video first if you prefer a visual guide before diving into the written steps! Step 1: Create a New Rule Set   In the SAS Decision Builder interface, click on ‘New Item’ and select ‘Rule Set’. Name your rule set according to the purpose it serves (e.g., ‘Eligibility_Check’ or ‘Approval_Criteria’). Once you’ve named your rule set, open it to begin configuring variables and adding rules.     Step 2: Define Variables for the Rule Set   Variables are essential for defining the data inputs and outputs in your decision logic. Navigate to the Variables Tab: In the rule set editor, go to the Variables tab. Add New Variables: Click on ‘Add Variable’ and name each variable (e.g., STATUS , BAD , JUSTIFICATION , OFFER_RATE ). Specify whether each variable is an Input (data coming from an external source) or an Output (data generated as part of the decision outcome). Assign the appropriate data type to each variable (e.g., integer, character, decimal). Defining variables first provides a clear structure for creating rules and avoids potential validation issues.     Step 3: Add Rules to the Rule Set   Now that you’ve defined the variables, it’s time to create the rules that will drive your decision-making process. Return to the Home Screen: Go back to the main screen of the rule set editor. Add a New Rule: Click on ‘Add Rule’ to create the first rule. Example Rules: Initial Status Assignment: Set STATUS to ‘REVIEW’ initially, indicating that all applications start in review. Denial Criteria: Create an IF condition to automatically deny certain applications. IF BAD = 1 (indicating bad credit history), THEN set STATUS to ‘Denied’, AND set JUSTIFICATION to ‘Automatic Denial Criteria Met’. Additional Denial Conditions: Add more OR and AND conditions to capture other scenarios that should result in denial. For example, conditions based on debt-to-income ratio or credit score thresholds. Approval Criteria: Define approval rules for applications meeting all eligibility criteria. For example, if an applicant’s credit history is clean and their loan request is within acceptable limits, set STATUS to ‘Approved’. Calculate Offer Rate for Approved Applications: If the application is approved, calculate OFFER_RATE based on a predefined variable like Rate_Low . This rule can adjust the offer rate dynamically, providing a customized rate for each approved applicant.     Step 4: Add the Rule Set to a Decision Flow   Now that your rule set is complete, you can add it to a decision flow to automate the decision process. Open the Decision Flow Editor: In SAS Decision Builder, navigate to the decision flow where you want to use the rule set. Add the Rule Set to the Flow: Right-click on the Start decision step in the flow. Select ‘Add’, then choose ‘Rule Set’. Choose the rule set you created (e.g., ‘Eligibility_Check’). Review and Configure Flow Variables: Add variables to the decision flow to ensure that data flows correctly into and out of the rule set. Mark each variable as Input or Output, depending on whether it’s coming from an input dataset or generated as part of the decision outcome.       Step 5: Run the Decision Flow and View Results   With the rule set added to the decision flow, you’re ready to test it and view the outcomes. Run the Decision Flow: Click Run in the Decision flow editor. A dialog will prompt you to select an input dataset. Choose the dataset with relevant data (e.g., loan applications). Specify a destination in your lakehouse where the results will be saved. Review the Results: After running the decision flow, navigate to the lakehouse table to view the decision outcomes. The results will show how each application was processed, displaying statuses (Approved, Denied, or Review) and justifications based on your rule set.   With these steps, you’ve successfully created and configured a rule set in SAS Decision Builder, added it to a decision flow, and viewed the outcomes. This process allows you to automate complex decision-making, making it more efficient, consistent, and data-driven.   Explore additional options in SAS Decision Builder to further refine your decision flows, adjust rules as business needs change, and incorporate advanced logic. Watch this guide to build a complete decision flow. ... View more

  1. General Questions   What is SAS Decision Builder?   SAS Decision Builder is a powerful SaaS product designed to help organizations automate and optimize complex decision-making processes. It enables users to create, deploy, and manage decision flows using business rules, Python code, and machine learning models, allowing for seamless integration of automated decisions into business workflows.   Who should use SAS Decision Builder? SAS Decision Builder is ideal for data scientists, business analysts, decision-makers, and IT professionals who need to automate repetitive decision-making processes. It is particularly valuable for industries like finance, healthcare, retail, and manufacturing, where decisions need to be made at scale with precision and compliance.   What are typical use cases for SAS Decision Builder? Common use cases include: Loan approval automation Fraud detection and prevention Customer segmentation and personalized offers Risk assessment and compliance checks Claims processing in insurance Inventory management in retail   2. Getting Started   How do I access SAS Decision Builder? SAS Decision Builder is available on the Microsoft Fabric Workload Hub. After subscribing, you can access it via the Microsoft Fabric environment.   What are the prerequisites for using SAS Decision Builder? SAS Decision Builder requires: To get started, you just need access to the Microsoft Fabric environment and a valid Fabric license to access SAS Decision Builder on the Microsoft Fabric Workload Hub Basic knowledge of decision automation, rules-based systems, and familiarity with Python (for advanced customizations)   What kind of training or resources are available for new users? The Learn and Support page for SAS Decision Builder offers tutorials, documentation, and video demos to help you get started. Additionally, SAS provides webinars and support options for users looking to deepen their understanding of the platform.   3. Features and Functionality   Can I use machine learning models in my decision flows? Yes, SAS Decision Builder allows you to integrate machine learning models, including those built in Azure ML Model, as part of your decision flows. You can combine rule-based logic with predictive models to enhance decision accuracy and automation.   What types of decision logic can I implement in SAS Decision Builder? You can implement a variety of decision logic, including: Rule sets for rule-based decisions Python code files for custom logic Branching logic to handle different decision paths Integration of machine learning models for predictive insights   How do I create and manage variables within SAS Decision Builder? Variables can be created in the Variables tab of each rule set or Python code file. You can define input and output variables, set data types (e.g., decimal, integer, string), and configure whether they’re used as inputs or outputs in the decision flow.   Can I modify rules or decision logic on the fly? Yes, SAS Decision Builder supports on-the-fly updates to rules and decision flows. You can easily edit existing rule sets, add or modify variables, and update logic without needing to recreate the entire flow.   What is the validation feature, and why is it important? The validation feature checks your decision flow for potential errors or configuration issues before you run it. Validation ensures that all rules, variables, and connections are set up correctly, reducing the risk of errors when executing your decision flow. 4. Technical and Deployment Questions   Where does SAS Decision Builder store decision outcomes? Decision outcomes can be stored in a designated lakehouse table within your Microsoft Fabric environment, where they can be accessed and analyzed for further insights.   What are the technical requirements for deploying SAS Decision Builder? SAS Decision Builder requires a Microsoft Azure subscription and access to Microsoft Fabric for deployment. It also integrates with other Azure services, such as data storage solutions, if needed.   How can I integrate SAS Decision Builder with other enterprise applications? SAS Decision Builder offers flexible integration options, including API-based integrations and connections to data sources within Microsoft Fabric. You can connect it with CRM, ERP, and BI systems to synchronize data and decisions across platforms.   Is SAS Decision Builder a low-code/no-code solution? SAS Decision Builder offers a combination of low-code and code-based options. Business users can set up rules and decision flows using a visual interface, while technical users can add custom logic using Python code files and machine learning models for advanced capabilities.   5. Troubleshooting and Support   I encountered an error during validation. How do I resolve it? Validation errors are displayed in the Errors panel. Click on each error to navigate directly to the component causing the issue, where you can make the necessary adjustments. Refer to the documentation for common error types and solutions.   What should I do if my decision flow produces unexpected outcomes? Review your rules, variables, and branching logic to ensure they are configured correctly. Use the validation tool to catch any setup errors and consult the Learn and Support page for troubleshooting tips.   How can I access support for SAS Decision Builder? SAS provides a variety of support options for SAS Decision Builder users: Documentation on the SAS website Video tutorials and product demos Community forums for support Technical support via SAS if you have a support contract   Can I request additional features or customizations for SAS Decision Builder? SAS values user feedback and considers feature requests in future updates. You can submit feature requests through SAS Decision Builder communities page.   6. Security and Compliance   Is SAS Decision Builder compliant with data protection standards? Yes, SAS Decision Builder is designed to comply with major data protection standards and can be deployed in environments that meet industry security standards. Additionally, it inherits the security and compliance capabilities of Microsoft Fabric.   How secure is the data processed by SAS Decision Builder? SAS Decision Builder uses Microsoft Azure’s security framework to protect data at rest and in transit. Access control, encryption, and data masking options are available to help safeguard sensitive information.   Can I control access to specific decision flows within SAS Decision Builder? Yes, SAS Decision Builder supports role-based access control, allowing you to manage user permissions and restrict access to certain decision flows, rule sets, or datasets as needed.   7. Pricing   How is SAS Decision Builder priced?   SAS Decision Builder pricing is based on a usage model and on number of seats. Usage metrics are based on the number of decisions that made using SAS Decision Builder, so each transaction sent to SAS Decision Builder is monitored and recorded against your plan. Details about SAS Decision Builder plans can be found on the Marketplace.   8. Product Roadmap and Future Enhancements   Are there plans to add more features to SAS Decision Builder? SAS continuously improves SAS Decision Builder based on user feedback and market trends. Future updates may include additional integrations, advanced analytics capabilities, and enhancements to the user interface. Stay tuned for announcements on upcoming releases.   How can I stay informed about product updates? Subscribe to SAS Decision Builder communities page to receive updates on new features, technical support, and product announcements for SAS Decision Builder. ... View more

SAS Information Catalog excels at organizing and understanding both data and analytic assets within your organization. Here's how it achieves this:   1. Data Cataloging: Automated Discovery: Agents crawl various data sources like files, tables, databases, and SAS libraries, automatically pulling in essential metadata about each asset.   SAS Information Catalog - Agents to Discovery data assets automatically     Discovery Agents Configuration and Execution Asset Summary   Agent Configuration - Type of Catalog Analysis Mode and Options       Advanced Analysis Options       Filter tables from a data source or schema from catalog process       Catalog and Analysis - Last Execution Summary         Metadata Enrichment: Users and tools can further enrich this metadata with details like descriptions, tags, lineage information, and data quality metrics. Coming soon - Glossary and Business Terms.   Catalog and Data Analysis Table Overview       Adhoc Tags and Search       Tag search with results based on assigned or user created tags         Standardization & Governance: The catalog establishes a single source of truth for data definitions, promoting consistency and adherence to data governance rules using auto generated dataset    Auto generated dataset summary           2. Analytic Asset Cataloging: SAS Viya Integration: Information Catalog seamlessly integrates with SAS Viya tools, automatically capturing metadata for models, reports, data flows, and other analytic creations.   Cataloged Assets at a Glance - Data and Analytic Assets available to discover by multiple user personas in an organization         Lineage Tracking: You can track different types of assets and their relationships with other assets, providing context for how analytical workflows operate.   End-to-End entity level lineage from Data to Analytic Models             3. Search for Discovery: SAS Information Catalog offers powerful search functionalities empowering users to find the desired assets with ease:   Keyword & Full-Text Search: Search across asset names, descriptions, tags, and even within data files for specific keywords or text patterns. Faceted Search: Filter results based on various criteria like data type, source system, creation date, usage statistics, or quality metrics. Natural Language Search: Formulate inquiries in natural language, and the catalog will intelligently interpret and return relevant results. Relevance Ranking: Search results are ranked based on factors like keyword match, user preferences, and asset usage, prioritizing the most relevant items.   Faceted Search based on specific dataset characteristics and functionalities         Natural Language Search that matches various attributes of a dataset to provide relevant results to end users       This search experience empowers users across the organization, from data analysts to business stakeholders, to efficiently discover and understand the data and analytic assets they need to make informed decisions. Additionally, advanced features like data lineage visualization provide deeper insights into how data flows through your organization and supports various analytic processes.   Remember, SAS Information Catalog offers various user interfaces and integration options that allow seamless access to the cataloged assets. SAS Information Catalog streamlines data and analytic asset cataloging and makes them readily discoverable through its intuitive search functionalities. ... View more

Watch this Ask the Expert session to learn the importance of data catalog and other data governance areas to improve delivery of analytic insights.    Watch the webinar   You will learn: Some of the common challenges hindering or decreasing the analytic pipeline delivery and business benefits. How to get reliable, trusted data quicker to maximize benefits from the data for reporting or analysis. How to integrate open source products.   The questions from the Q&A segment held at the end of the webinar are listed below and the slides from the webinar are attached.   Q&A  Are there any options to integrate or share metadata with other applications?  I did not go there, but thanks for asking the question. One thing I should mention is yes, we are working on that, and we currently have a way to do that as well. We have integrated with this open-source metadata standard called Egeria. Egeria is part of the Linux Foundation. It's an open metadata standard, more like a metadata marketplace, where different application vendors can share their metadata. It benefits the organization, so you do not have to go and stitch all these together, right? Right now, we are contributing our metadata from the SAS information governance product to Egeria. If the data stored in the organization has enabled, it automatically events all the assets that the user sees in the product to Egeria, which is open metadata standard and can be combined with other application metadata so that way they can see this lineage across different vendors. Obviously, we are also looking at integrating with open lineage and markets, which is other open metadata standard tools, to augment the lineage capability. We wanted to address the analytic use case with the discovery and lineage and be able to partner and integrate with other tools as well, whether it is open source and creating this innovation and build a partnership. The simple answer is yes, we can. This material is available to extract to the third-party and we are also able to take the third-party metadata into SAS using rest APIs and building that Azure pipeline. We build on top of that by integrating with the other vendors.    What role does the cataloged data play in Lineage?  The group, but I should also mention that we do not want to see lineage as a separate piece. That is one of the key things we are carrying out as part of this discovery. The admin is not just building the discovery Cadillac for users to identify datasets, they are also in the background. We are building the lineage meaning. As the data has been cataloged, we also capture the dataset usage. Whether the dataset used in the report or the model, everything is captured. The relationship between one dataset to another asset is being captured as part of that cataloging as well. What that means is when a user discovers the asset here, they could also go to lineage and discover the asset as well. Same exact asset, but more towards the left or right discovery of lineage.    Where can I find this Catalog Home? I cannot see it under SAS Drive. Is it something that our organization needs to activate?  If you look at the side menu bar, discover information assets. That is the SAS information governance and catalog the product. This is the discovery of information assets.    Can SAS Information Catalog discover CAS Lib and Compute Lib?  If you are existing SAS users, you are aware of the SAS library and CAS library where you could look at SAS datasets. It does both. In fact, all the data set catalog that SAS table and CAS table. This is, you know, a traditional SAS table with SAS7BDAT format. I can look at that individually or CAS table, which is the SASHDAT for the in-memory engine that can be cataloged as well. But both can be available together for the users.    Does this platform run on prem and or in the cloud?  This is native cloud deployment whether private or public cloud. If there is an on Prem cloud private cloud, you could deploy this as well. Can we deployed on multiple public clouds, although we have partnered with Azure for better integration. This is a cloud deployment, and it is multi cloud supported.    Is information governance part of SAS Viya or should it be licensed additionally?  That is important as you start looking at using this. The information catalog product packaging wise is any visual package or bundles that, as part of SAS Viya, will get the basic version of the catalog product. Basic version would have the search and be able to explore and see some of the profile metrics. That is any visual SAS Viya products along with the SAS Visual Analytics. SAS models would contain the basic version of catalog because regardless of whether they buy the reporting product or analytics product, they would want to discover it. We want to give the ability of the basics as information. But they can add additional capabilities, such as the semantic Identification or Information privacy. There are others, such as machine learning, business summary identification, and language detection and workload. Those are part of the sales information governance license. The basic is included with all the bundles, which is the sense information catalog. The license is the SAS information governance with advanced features.    Can cloud data be accessed?  Yeah, of course. Given this is a cloud-native deployment application, regardless of what cloud it is. You can easily call any cloud data on Azure, whether SQL Azure Synapse or Azure blob store. Same goes for AWS - AWS S3, Redshift or Yanmar Yanmar. Amazon's Hadoop version symbols of GCP for Google, as well. Any cloud data can be crawled and cataloged and be made available for discovery.    What features are available to help with the consumption of these data assets? For example, is there a way to combine columns from different sources into a logical view for consumption by applications?  The simple answer is yes and that's part of the action. When we were exploring the data assets in the action part, we have multiple actions. Prepare data is the action use can take while using SAS Information Governance to do data preparation. One data could be coming from Snowflake in the cloud and other data could be coming from a data warehouse sitting in the on premise. Regardless of where it is and if the connector is established, all kinds of transformation could be applied. Column can be removed, or column can be transformed. All of those preparations can be done at the prepared data layer and you can create this analytic based table, or one table more for preparation by the data engineer (or in some cases the analyst).    Does information governance allow for metadata updates by various stakeholders in a decentralized manner or is it more of a centralized Data management method?  If integration with other metadata sources, 3 rd party vendors and/or sharing metadata was meant by decentralized then yes, it is. SAS Viya can share metadata with other players using integration with Egeria open metadata standard.    Recommended Resources SAS Information Governance page Getting started with SAS Information Catalog in SAS Viya SAS Information Governance Release Highlights Please see additional resources in the attached slide deck.   Want more tips? Be sure to subscribe to the Ask the Expert board to receive follow up Q&A, slides and recordings from other SAS Ask the Expert webinars.   ... View more

Presenters Kumar Thangamuthu, SAS; Nancy Rausch, SAS Abstract SAS continues to enhance its data management functionality to provide a comprehensive and integrated set of capabilities for collecting, transforming and managing your data. New features include enhanced capabilities for working with data from a wide variety of environments and types, including Hadoop, cloud environments, RDBMS, files, unstructured data, streaming and others. There are new automation and discovery enhancements, as well as new support for building flows and cataloging assets to understand the context and governance of useful data in an enterprise. Watch the presentation Watch What’s New in SAS® Data Management on the SAS Users YouTube channel.   ... View more

The popularity of cloud data storage has grown exponentially over the last decade as more and more organizations are transitioning from on-premises to cloud data storage and data management. Microsoft Azure is one of the big players accelerating the move to cloud. In this article, we will look at the overview of SAS® to access data in Azure Storage and Big Data.   As organizations start to realize the impact of digital transformation, they are moving storage to the cloud as they move their computing to the cloud. Data Storage in the cloud is elastic and responds to demand while only paying for what you use, similar to compute in the cloud. Organizations must consider data storage options, efficient cloud platform and services, and migrating SAS applications to the cloud.   Organizations can connect from the SAS platform and access data from the various Azure data storage offerings. Whether it is Azure Data Lake Storage or Big Data HDInsight, SAS/ACCESS engines and data connectors have these covered with optimized data handling abilities to empower organizations going through digital transformation journey.   SAS AND AZURE DATA LAKE STORAGE SAS Viya can read and write ORC and CSV data files to Azure Data Lake Storage Generation 2(ADLS2). There is a new data connector called SAS ORC Data Connector to facilitate the data transfer between CAS and ADLS2. The SAS ORC Data Connector enables you to load data from an Apache Optimized Row Columnar table into CAS. This data connector can be used with a path or Azure ADLS2 CASLIB.   SAS LIBRARY TO ADLS /*create a CAS session */ cas casauto; caslib "Azure Data Lake Storage Gen 2" datasource=( srctype="adls" accountname=”sasdemo” filesystem="data" dnsSuffix=dfs.core.windows.net timeout=50000 tenantid=<Azure Application Tenant ID UUID> applicationId=<Azure registered application UUID> ) path="/" subdirs global /* creates library reference for SAS compute */ libref=AzureDL; caslib _all_ assign; Here is an explanation of the parameters that are used to create a caslib: CASLIB – A library reference. The caslib is the space holder for the specified data access. The Azure Data Lake Storage Gen 2 CAS library is used to specify the ADLS data source. SRCTYPE – Source type is ADLS which corresponds to Azure Data Lake Storage connection. ACCOUNTNAME – Azure Data Lake Storage account name. FILESYSTEM – ADLS container file system name. TENANTID & APPLICATIONID – Available from the Azure Registered Application page for your organization or individual use. PATH – Points to the directory structure where the file system resides. LIBREF – Creates a SAS library reference along with a CAS library. SAS Viya uses the available CAS session CASAUTO to create the CAS library reference to ADLS. In this example, it uses a CAS clustered environment with 3 nodes, including 2 worker nodes. To make the CAS library available to all the users, global parameters can be used. ORC or CSV can be loaded from the ADLS blob container file system “data” to a CAS in-memory cluster or saved to ADLS from CAS.   LOAD AND SAVE ORC DATA TO AZURE STORAGE FROM SAS VIYA Let’s look at an example to load a SAS dataset to a CAS in-memory server. Once the data is in CAS, it can be used for any distributed data processing, report, analytics, or modeling. The final CAS in-memory data output is saved as an ORC file to Azure Data Lake Storage. proc casutil; load data=sashelp.class casout="class" outcaslib="adls" replace; save casdata="class" casout="class.orc" replace; quit; Important parameters: PROC CASUTIL – CASUTIL procedure that works with CAS tables including data transfer, table & file information, and drop & delete tables. LOAD – CAS action set to load data to CAS in-memory servers DATA – SAS traditional data set used as input CASOUT – Output CAS in-memory distributed table OUTCASLIB – Output CAS Library reference SAVE – CAS action set to save CAS in-memory table to ADLS CASDATA – Input CAS table CASOUT – ORC output file saved to ADLS   SAS AND AZURE HDINSIGHT HADOOP DATA LAKE           Traditionally, the Hadoop platform has been deployed to solve an organization’s Data Lake business needs on-premise. With more and more incoming data along with deploying analytical applications such as SAS Viya in the cloud, organizations are moving some of their Data Lake needs to the cloud along with the data. Azure HDInsight is one such Hadoop service that provides a cloud native Hadoop platform along with elasticity to scale up or down the Data Lake. The SAS platform can leverage Azure HDInsight Hadoop service to access, load, or save data using the SAS Access to Hadoop engine.   LIBNAME STATEMENT TO CONNECT AND ACCESS DATA option set = SAS_HADOOP_CONFIG_PATH = "/opt/hdinsight/conf"; option set = SAS_HADOOP_JAR_PATH = "/opt/hdinsight/jars"; options sastrace = ',,,sd' sastraceloc = saslog no$stsuffix fullstimer; libname hdilib HADOOP uri='jdbc:hive2://sgfdemo.azurehdinsight.net:443/default;ssl=true?hive.server 2.transport.mode=http;hive.server2.thrift.http.path=hive2' server='sgfdemo.azurehdinsight.net' user="admin" pw="demopassword" schema="default"; As a one-time configuration setup, before creating a SAS library reference and connecting to an Azure HDInsight cluster, the Hadoop administrator would execute a HadoopTracer Python script available from SAS to extract all the necessary JAR and config files. The SAS administrator can further use these extracted files to connect to an Azure HDInsight cluster by setting two option variables. This step is performed regardless of the type of Hadoop distribution or cloud platform.   Some of the important parameters: SAS_HADOOP_CONFIG_PATH - Specifies the directory path for the Azure HDInsight cluster configuration files. SAS_HADOOP_JAR_PATH - Specifies the directory path for the Azure HDInsight JAR files. URI – Azure HDInsight JDBC URI to connect to Hive server 2. Once a similar JDBC URI is retrieved from Azure HDInsight documentation, just modify the HDInsight server name. The JDBC URI contains some of the necessary parameters that are enabled and assigned values by default. SSL is set to true by default, and REST transport mode is set to HTTP. Data can be loaded from an Azure HDInsight cluster to the SAS platform or saved to the cloud. SAS PROCs such as Proc Append, Sort, Summary, Means, Rank, Freq, and Transpose are supported on Azure HDInsight cluster. Furthermore, the DATA step and PROC SQL data preparation with bulk load are handled to save data efficient on the cloud.   SAS can access data directly from Azure Storage and Big Data or through other Azure intermediary data sources that uses Azure ADLS2 or HDInsight as the persistent storage layer. As the end users migrate from on-premise to cloud, they also asses the data storage landscape instead of just lifting and shifting to similar data source in the cloud to address Self-Service data management. Variety of data types such as structured, unstructured and the need to access data faster are some of the key deciding factors to choose cloud blob or big data storage options. ... View more

Many common data platforms already in use are being refactored and delivered as service offerings to Azure cloud customers. Azure offers database service technologies that are familiar to many organizations. It is important to understand the terminology and the different database services to best meet the demands of your business use case or application. Benefits to organizations are reducing hardware and software footprint to manage. Databases that scale automatically to meet business demand and software that optimizes and creates backups means organizations can spend more time deriving insights from their data and less time managing infrastructure.   SAS provides efficient SAS Data Connectors and SAS In-Database Technologies support to Azure database variants. A data storage running in Azure cloud is much like an on-premise database, but instead Microsoft manages the software and hardware. Azure services can take care of the scalability and high availability of the database with Database as a Service (DBaaS) offerings and minimal user input. SAS integrates with Azure cloud databases whether SAS is running on-premise or in the cloud.   SAS AND AZURE SQL DATABASE          Organizations can connect and access data from an Azure SQL Database using SAS/Access to Microsoft SQL Server from SAS. All the features from SAS/Access to SQL Server running on-premise would be available in the cloud as well. Running the SQL database in the cloud gives organizations elasticity to scale the database. Let’s look at code samples to connect and access data from using SAS library and CAS library.   LIBNAME STATEMENT TO CONNECT TO AZURE SQL DB           Azure SQL Database connection information typically specified in the odbc.ini file on the SAS servers, along with a data source name (DSN). Specify the DSN name in the libname connection parameter.   libname mydblib sqlsvr noprompt="uid=sgfdemo; pwd=demopassword; dsn=Azure SQL Database;" stringdates=yes; The SQLSVR connection parameter enables the LIBNAME statement to use SAS/Access to SQL Server to connect to Azure SQL Database.   SAS VIYA CAS LIBRARY TO CONNECT TO AZURE SQL DB          Similarly, SAS/Access to SQL Server can be used to connect to an Azure SQL Database using a data connector. This approach is helpful to organizations with streamlined data movement when SAS Viya is running in the cloud to load and save data between Azure SQL Database and SAS Viya in-memory servers.   cas casauto; caslib azsqldb desc='Microsoft SQL Server Caslib' dataSource=(srctype='sqlserver' username='sas' password='demopassword' sqlserver_dsn="Azure SQL Database" catalog='*'); Azure SQL Database connection information has to be added to the odbc.ini files in all the CAS cluster nodes. This facilitates optimized data movement between multiple CAS worker nodes and Azure SQL Database.   SRCTYPE – Source type SQLSERVER is the key parameter to connect to an Azure SQL Database using SAS Viya Data Connector to Microsoft SQL Server.   Load and Save SAS Dataset to Azure SQL DB           With one table in the Azure SQL Database, let’s load another table from a SAS Viya in- memory cluster to the Azure SQL Database. In the following code sample, first we are loading a SAS data set named cars to a CAS in-memory cluster. Then the distributed CAS in-memory table is saved to the Azure SQL Database.   proc casutil; load data=sashelp.cars outcaslib="azsqldb" casout="cars" replace; save casdata="cars" casout="cars" replace; quit; One parameter to mention here, OUTCASLIB, contains the name of the CAS library that we created earlier to access the Azure SQL Database. Information about other parameters has been furnished in the ADLS section above.   SAS VIYA AND AZURE MYSQL DATABASE           MySQL is another widely used relational database management system that is accessible from SAS Viya as an Azure MySQL database with elastic server configurations to support various data volumes. SAS Data Connector to MySQL can be used to access, load, and save data between CAS in-memory cluster nodes and an Azure MySQL Database.   Other flavors of MySQL databases such as Azure MariaDB Database can also be accessed using SAS Data Connector to MySQL. Steps are exactly the same as MySQL, just modify the server host to Azure MariaDB, along with the appropriate credentials.   cas casauto; caslib azurems desc='MySQL Caslib' dataSource=(srctype='mysql' host='sgfdemo.mysql.database.azure.com' username='sas@sgfdemo' password='demopassword' database="demo") /* global */ ; The MySQL client must be installed on all the CAS cluster nodes. The MySQL client library path must be added to the CAS configuration settings file on the nodes.    SRCTYPE – Source type MYSQL is the key parameter to connect to an Azure MySQL Database using SAS/Access to MySQL.   LOAD AND SAVE SAS DATASET TO AZURE MYSQL DATABASE proc casutil; load data=sashelp.cars outcaslib="azurems" casout="cars" replace; save casdata="cars" casout="cars" replace; list files; quit; SAS Viya uses SAS Data Connector to MySQL to save a CAS in-memory table named cars to an Azure MySQL Database. The Save CAS action, which is part of the TABLES CAS action set has been called to perform a serial save in this case.   SAS VIYA AND AZURE POSTGRESQL DATABASE         The SAS platform can access data from another popular relational database management system(RDBMS), PostgreSQL. Azure PostgreSQL Database is a Database as a Service (DBaaS) offering to support data movement as needed through various server configurations based on data volume and demand.   SAS LIBRARY TO LOAD SAS DATASET libname azurepg postgres server="sgfdemo.postgres.database.azure.com" port=5432 user=sas@sgfdemo password='demopassword' database=postgres; /*Load a SAS dataset to Azure PostgreSQL database*/ data azurepg.cars; set sashelp.cars; run; LIBNAME statement uses the POSTGRES keyword to utilize SAS/Access Engine to PostgreSQL to connect to an Azure PostgreSQL Database. In this example, a SAS dataset named cars is loaded to the destination cloud PostgreSQL database.   CAS LIBRARY TO AZURE POSTGRESQL cas casauto; caslib postgrescaslib desc='PostgreSQL Caslib' dataSource=(srctype='postgres' server='sgfdemo.postgres.database.azure.com' username='sas@sgfdemo' password='demopassword' database="postgres"); The CAS library postgrescaslib is created by connecting to the destination Azure PostgreSQL Database. Source type POSTGRES is the keyword that utilizes SAS Data Connector to PostgreSQL.   Load and Save SAS Dataset to Azure PostgreSQL DB proc casutil; load data=sashelp.class outcaslib="postgrescaslib" casout="class" replace; save casdata="class" casout="class" replace; list files; quit; SAS Viya uses SAS Data Connector to PostgreSQL to save the CAS in-memory table named cars to the Azure PostgreSQL Database. The Save CAS action, which is part of the TABLES CAS action set has been called to perform a serial save in this case. The LIST FILES action in the CASUTIL procedure show the cars table has been saved to the database in the cloud.   Once the data is loaded to CAS in-memory servers for distributed parallel processing or SAS compute server, advanced analytics and model methods can be applied to get relevant insight out of the prepared data. Whether the final decision is to keep data on-premise or in the cloud or a hybrid approach, SAS Data Connector supports virtually any data sources to access, load, move, model data efficiently, or navigate through the analytic life cycle. It is important to think about the use case for your database and the type of data you plan to store before you select an Azure database service. ... View more

  Modern Data Lake The data lake idea focuses on storing all analyzable data sets in raw or lightly processed form into the easily expandable scale-out Hadoop infrastructure to ensure that the fidelity of the data is preserved. The modern data lake will be a “managed” data lake, meaning one that uses a data lake management platform to manage data ingestion, apply metadata and enable data governance so that users know what’s in the lake and can use the data with confidence. These factors help to avoid data lake turning in to a “data swamp”.   Modern managed data lakes allow businesses to explore their data quickly and easily, identifying opportunities for business and process improvements across the organization as well as to better understand their customers. Cybersecurity and personal data protection play vital role in every organization. This is a very big data problem as threats come from internal and external sources of an organization. The General Data Protection Regulation (GDPR) is intended to standardize expectations and protect personally identifiable information on employees, clients, and applicable data subjects. This means cybersecurity data collection and analysis has to become proactive and always on. Data governance is important in securing the data in data lake with organization’s procedures and policies that manage the data usage, availability, quality, privacy and security of data.   Data Lake Market Growth   Streaming data from high-velocity, high-volume sources from small to large companies contributing to explosive growth of data lake market. Streaming data can be   Constantly generated, in small bursts like IoT, product logs Accumulated quickly – Billions of records occupying hundreds of petabytes Unstructured or semi-structured in its original form   Data Lake growth drivers   Evolution of data lakes on cloud such as Amazon S3 or Azure data lake store that can offload the maintenance effort, take care of auto-scaling, encryption and ease of management Availability of choice and better accessibility with the emergence of data lake querying tools based on SQL and ad-hoc querying Varied pricing options for agile businesses, especially with the separation of storage and compute methodologies in data lake. Desire to give data scientists access to enterprise data for exploration, discovery, and insight creation   Effect of latest trends on Data Lake   Trends like IoT, AI, live streaming data have created enormous amount of data being thrown into data lakes. Though managing a data lake or data hub to derive business insights used to be expensive, moving data lakes to the cloud have eased some of those pains, along with elastic infrastructure. Cloud-based data lakes tend to offer better availability than what is guaranteed on-premises, faster time to value and time to deploy for new projects, data sources and applications already cloud-based and scalable. Data Lakes are evolving to handle different modes of data movement (streaming for example) and different formats of data that are either consumed or generated by the IoT or AI projects.   Data Lake Vs Data Warehouse   Data moves through a data lake much faster than a data warehouse, reducing latency and providing faster time to analyzed data. Data lakes uses “schema-on-read” approach, accelerating insights and lowering the cost of acquiring new types of high-volume data, sensor data, social media data and clickstream data. Data in data lakes are continuous and not confined to periodic updates, as with data warehouses, allowing for real-time analysis from streaming data. Though there are significant differences between data lakes and data warehouses, they both complement each other in an enterprise environment. A data lake becomes a landing and staging area for the data warehouse and other downstream applications. Meanwhile, data warehouse serves as a repository of integrated, historical data that is dimensionally modelled and can be queries in an ad hoc manner by concurrent business users. Data Lakes provide flexibility in the ability to apply structure to data after it has arrived in the Data Lake. Data warehouses often demand well-defined structures up front into which data is fit.   Limitations of Data Lake   Data lakes lack strong governance, metadata, security and other standard features of enterprise software, though managed data lakes are starting to address some of these limitations. As companies move beyond piloting advanced analytics projects to run data lake in production and at scale, they have found the software tools quite expensive, they also require investments in people and products required to manage the open source functionality and infrastructure.   Data Lake play in industry verticals   The basic nature of the data lake concept is industry agnostic. A data lake is a storage system that can store large amounts of data in its original format until required by advanced analytic and visualization applications to derive insights. Verticals simply differ by how they use the data, in other words, how the data is transformed to be used by a specific organization or industry.   For example, government initiatives like Smart Cities, that can manage traffic, optimize power grids, enhance education systems, track vehicle patterns, connected vehicles and much more generate massive amounts of data. Data lakes and analytics can assist cities in building more sustainable, livable environments.   Banking industry is also moving fast towards making the most of growing and monetizing data. Entering the big data universe, banks have now come to rely on data lakes to bring all the data together and accelerate the data-to-action cycle. Banks accelerate the pace to go from data to insightful reports using Hadoop-based data lake synched with visualization and analytics tools.     SAS & Data Lake           SAS is a leader in big data management and advanced analytics solutions that help customers make better decisions, faster. The following capabilities, all supported by SAS, can enable data management and analytics for data lake and other valuable repositories:   Data integration - Data movement, in-database processing, and native data access to traditional and emerging data sources such as Hadoop.   Data quality - Cleanse, standardize, and enrich data in real time and batch with prebuilt rules.   Self-service big data preparation - Business users profile, cleanse, and transform data on Hadoop without writing code.   Business glossary and metadata management - Track lineage, business rules, descriptive details, and workflow for improved governance of your data assets.   Event stream processing - Analyze real-time streams of data in motion for better decisions.   Data virtualization - Provide blended, secure views of your data without moving it.   Hadoop support - Access, deliver, and process data inside Hadoop across both the data management and analytics life cycle.   Visualization and advanced analytics - Deliver cutting-edge visualization and analysis capabilities without requiring analytical skills. ... View more