In today’s data-driven world, organizations rely heavily on well-documented workflows for governance, compliance, and collaboration. However, manually documenting flows, such as those in SAS Studio, can be time-consuming and prone to human error. This is where automation steps in. By leveraging Azure OpenAI and a SAS Studio custom step, you can generate clear, detailed, and governance-focused documentation in seconds. Let’s explore how it works.   Watch the Demo Videos   Before diving into the details, watch this demo video showcasing how to document SAS Studio flows using a SAS Studio custom step calling an Azure OpenAI large language model.     This custom step allows you to document not only SAS Studio Flows, but also SAS and Jupyter notebooks, SAS code files and potentially other files containing code.       Required Components   To replicate this example, you’ll need the following:   SAS Viya with Python Ensure you have a working SAS Viya environment and Python installed and configured for the environment. The custom step uses a Python program. Python packages required: ‘python-dotenv’ and ‘requests’. Azure OpenAI Resource Create an Azure OpenAI resource and deploy a model (e.g., GPT-4o). .env File Create a configuration file named .env with the following content:   AZURE_OAI_ENDPOINT='https://my_endpoint.openai.azure.com/' # change my_endpoint AZURE_OAI_KEY='my_api_key' # change my_api_key AZURE_OAI_DEPLOYMENT='gpt-4o'   Display Hidden Files .env is a hidden file. To view it in SAS Studio, from Options > Preferences > General > check Display hidden files. System Message File The system_message.txt file shapes the documentation by acting as a guide for the model, defining the structure, level of detail, and focus areas (e.g., summaries, transformations, compliance) to ensure the output aligns with specific requirements. Create a file named system_message.txt with the following content:   You are an AI assistant specialized in documenting SAS Studio flows (flw files) for Governance and Compliance purposes. Your task is to analyze a SAS Studio flow, including its visual representation (image) and the underlying code, to generate detailed and precise documentation. Follow these steps: Summary: Start with a high-level summary of the SAS Studio flow. Include: The purpose of the flow. Key inputs (datasets or files used). Key outputs (datasets or files generated). A brief description of the transformations or processes applied. Step-by-Step Explanation: Break down the flow into individual steps and explain: The purpose of each step. Inputs and outputs for the step. Any transformations, joins, filters, or aggregations applied. Detailed Column Mapping Table: Create a table titled "Detailed Column Mapping for Each Step in the Flow". For each step: List all columns involved. Specify their names before and after the step. Highlight any changes applied to the columns (e.g., renaming, transformations, additions, deletions). Use the following table format: Step Column Name Changes (e.g., renamed, transformed, added, deleted) Description of Change (if applicable) Step Name/ID Column_Name_1 Renamed to New_Column_Name_1 Column renamed for consistency Step Name/ID Column_Name_2 Transformed Applied log transformation Step Name/ID Column_Name_3 Deleted Column removed as it is no longer needed Governance and Compliance Notes: Add a section at the end to highlight: Any potential compliance concerns (e.g., PII data transformations, data lineage issues). Suggestions for improving documentation or flow design for better governance.   Steps to Document   Step 1: Open SAS Studio   Launch SAS Studio and navigate to the Develop Code and Flows section.   Step 2: Create the Custom Step   In SAS Studio, create a custom step and paste in the Prompt UI and Program the content provided at the end of this post. Save it as LLM - Document Flows with Azure OpenAI.step.     Step 3: Create a Flow Using the Custom Step   From the Control Library, drag and drop the custom step titled LLM - Document Flows with Azure OpenAI.step into your flow canvas.   Step 4: Configure the Inputs   Fill in the required fields in the custom step configuration panel:   Select the flow to be documented (.flw file): Choose the .flw file you want to document (e.g., Car_Make_with_SubFlows.flw). Choose the folder where the .env file is stored: Provide the path to the folder containing your .env file (e.g., /azuredm/code). Pick the file where the LLM system message is stored: Provide the path to the system_message.txt file. Specify the output file name: The file where the documentation will be saved (e.g., Car_Make_with_SubFlows.txt).     Step 5: Run the Flow Using the Custom Step   Click Run to execute the custom step. The Azure OpenAI model will analyze the SAS Studio flow and generate detailed documentation based on the provided system message.   What ensures the accuracy of the documentation is that the entire .flw file is read and included as a single long string in the model prompt. This method allows the Azure OpenAI model to access the complete flow structure, dependencies, and logic, ensuring that no detail is overlooked. By providing the model with the full context, the documentation reflects the flow's exact operations, making it both precise and comprehensive.   Step 6: Review the Output   Locate the output file (e.g., Car_Make_with_SubFlows.txt) in the specified folder. Open the file to review the generated documentation, which includes:   High-Level Summary: A concise overview of the SAS Studio flow, including its purpose, key inputs, outputs, and transformations. Step-by-Step Explanation: A breakdown of each step in the flow, including its purpose, inputs, outputs, and transformations. Detailed Column Mapping Table: A table that documents changes to columns during the flow, such as renaming, transformations, or deletions. Governance and Compliance Notes: Highlights compliance concerns, data lineage, and suggestions for improvement.     The output is readable markdown, which can be opened with Visual Studio Code:         Custom Step Features   The custom step simplifies the documentation process with the following features:   Automated Analysis: Leverages Azure OpenAI to analyze flow components and generate detailed documentation. Governance Focus: Highlights compliance concerns, metadata, data lineage, and PII handling recommendations. Customizable System Message: Modify the system_message.txt file to tailor the documentation style and content.   Conclusion   In summary, this custom step provides a quick and efficient way to document SAS Studio Flows, notebooks, and other workflows. It brings clarity to complex or opaque processes, making it easier to understand transformations, data lineage, and compliance considerations. With a few adjustments, you can customize the output to suit your specific needs.   Custom Step Code   Prompt UI { "showPageContentOnly": true, "pages": [ { "id": "flowDoc", "type": "page", "label": "Document SAS Studio Flows with Azure OpenAI", "children": [ { "id": "text1", "type": "text", "text": "Inputs: Choose the flow to be documented (.flw file), the connection configuration for your Azure OpenAI model (.env file), the model system message, which must be stored in a 'system_message.txt' file.", "visible": "" }, { "id": "input_file", "type": "path", "label": "Select the flow to be documented (.flw file):", "pathtype": "file", "placeholder": "Car_Make_with_SubFlows.flw", "required": false, "visible": "" }, { "id": "env_file_folder", "type": "path", "label": "Folder where the .env file is stored", "pathtype": "folder", "placeholder": "/azuredm/code", "required": false, "visible": "" }, { "id": "messages", "type": "path", "label": "File where the LLM system message is stored", "pathtype": "file", "placeholder": "system_message.txt", "required": false, "visible": "" }, { "id": "text2", "type": "text", "text": "Output: choose the output file where the flow documentation is written (.txt file).", "visible": "", "indent": 0 }, { "id": "output_file", "type": "path", "label": "Write the output to (.txt file):", "pathtype": "file", "placeholder": "Car_Make_with_SubFlows_.txt", "required": false, "visible": "" } ] }, { "id": "help1", "type": "page", "label": "How To Use the Custom Step", "children": [ { "id": "text3", "type": "text", "text": "Custom Step: Document SAS Studio Flow\n\nPurpose:\nThis custom step processes a selected SAS Studio .flw file, sends its content to an Azure OpenAI endpoint, and generates documentation for the flow. The output is saved in a .txt file.\n\nInputs:\n1. Flow File (.flw): The path to the SAS Studio flow file to be documented.\n2. Environment File Folder (.env): The folder containing the .env file for Azure OpenAI configuration.\n3. System Message File (system_message.txt): A .txt file containing the system message for the OpenAI API.\n\nOutput:\n1. The .txt file where the documentation will be saved. The file contains Markdown-formatted text.\n\nHow It Works:\n1. The .flw file content is read and prepared for processing.\n2. A system message is read from the messages file.\n3. The .env file is loaded to retrieve Azure OpenAI credentials and endpoint details.\n4. The .flw file content and system message are sent to the Azure OpenAI API for processing.\n5. The API response, which contains the generated documentation, is saved to the specified .txt output file.\n\nSteps:\n1. Choose the inputs.\n2. Specify the .txt file to save the documentation.\n3. Run the step to generate the documentation.\n\n--- system_message.txt sample file --- \nYou are an AI assistant specialized in documenting SAS Studio flows (flw files) for Governance and Compliance purposes. Your task is to analyze a SAS Studio flow, including its visual representation (image) and the underlying code, to generate detailed and precise documentation. Follow these steps:\n\nSummary: Start with a high-level summary of the SAS Studio flow. Include:\nThe purpose of the flow.\nKey inputs (datasets or files used).\nKey outputs (datasets or files generated).\nA brief description of the transformations or processes applied.\nStep-by-Step Explanation: Break down the flow into individual steps and explain:\nThe purpose of each step.\nInputs and outputs for the step.\nAny transformations, joins, filters, or aggregations applied.\nDetailed Column Mapping Table: Create a table titled \"Detailed Column Mapping for Each Step in the Flow\". For each step:\nList all columns involved.\nSpecify their names before and after the step.\nHighlight any changes applied to the columns (e.g., renaming, transformations, additions, deletions).\nUse the following table format:\nStep\tColumn Name\tChanges (e.g., renamed, transformed, added, deleted)\tDescription of Change (if applicable)\nStep Name/ID\tColumn_Name_1\tRenamed to New_Column_Name_1\tColumn renamed for consistency\nStep Name/ID\tColumn_Name_2\tTransformed\tApplied log transformation\nStep Name/ID\tColumn_Name_3\tDeleted\tColumn removed as it is no longer needed\n\nGovernance and Compliance Notes: Add a section at the end to highlight:\nAny potential compliance concerns (e.g., PII data transformations, data lineage issues).\nSuggestions for improving documentation or flow design for better governance.", "visible": "" } ] } ], "values": { "input_file": "", "env_file_folder": "", "messages": "", "output_file": "" } }   Program /* Run the Python code within PROC PYTHON */ proc python; submit; # The following contains the Python Code to be written inside a PROC PYTHON. import os from dotenv import load_dotenv import requests # Get variables from SAS env_file_folder = SAS.symget('env_file_folder') input_file = SAS.symget('input_file') output_file = SAS.symget('output_file') messages = SAS.symget('messages') # Extract from SAS variables to resolve to Python paths env_file_folder = env_file_folder.replace('sasserver:', '') input_file = input_file.replace('sasserver:', '') output_file = output_file.replace('sasserver:', '') messages = messages.replace('sasserver:', '') print("env_file_folder:", env_file_folder) print("input_file:", input_file) print("output_file:", output_file) print("messages:", messages) # Folder where .env file is stored os.chdir(env_file_folder) def process_file(input_file, output_file): try: # Read the input file with open(input_file, 'r', encoding='utf-8') as f: content = f.read() # Print the length of the string to verify print(f"The length of the SAS Studio flow as a string is: {len(content)}") # Read LLM system message with open(messages, 'r', encoding="utf8") as file: system_message = file.read() # Get configuration settings load_dotenv() azure_oai_endpoint = os.getenv("AZURE_OAI_ENDPOINT") azure_oai_key = os.getenv("AZURE_OAI_KEY") azure_oai_deployment = os.getenv("AZURE_OAI_DEPLOYMENT") azure_oai_model = azure_oai_deployment api_version = '2024-05-01-preview' # this might change in the future # Request Header headers = { "Content-Type": "application/json", "api-key": azure_oai_key, } # Payload for the request payload = { "messages": [ { "role": "system", "content": [ { "type": "text", "text": f"{system_message}\n" } ] }, { "role": "user", "content": [ { "type": "text", "text": f"Document the following SAS Studio flow. FLW file content: --- {content} ---" } ] }, ], "temperature": 0.5, "top_p": 0.9, "max_tokens": 2500 } ENDPOINT = f"{azure_oai_endpoint}openai/deployments/{azure_oai_model}/chat/completions?api-version={api_version}" # Send the request try: response = requests.post(ENDPOINT, headers=headers, json=payload) response.raise_for_status() # Will raise an HTTPError if the HTTP request returned an unsuccessful status code except requests.RequestException as e: raise SystemExit(f"Failed to make the request. Error: {e}") # Handle the response as needed (e.g., print or process) response_data = response.json() # Extract the text content # This will vary based on the APIs JSON structure text_content = response_data['choices'][0]['message']['content'] print("\n Response: \n" + text_content + "\n") # Write the response to a file with open(output_file, mode="w", encoding="utf8") as results_file: results_file.write(text_content) print(f"\nResponse written to {output_file}\n") except Exception as e: error_message = f"Error: {e}" print(error_message) # Pass the error message back to SAS log SAS.submit(f'data _null_; put "{error_message}"; run;') # Run the processing function try: process_file(input_file, output_file) except Exception as e: error_message = f"Error: {e}" print(error_message) # Pass the error message back to SAS log SAS.submit(f'data _null_; put "{error_message}"; run;') endsubmit; run;   Thank you for your time reading this post. If you liked the post, give it a thumbs up! Please comment and tell us what you think about the Visual Studio Code Extension. If you wish to get more information, please write me an email.     Find more articles from SAS Global Enablement and Learning here. ... View more

The integration of conversational AI with decision-making systems is revolutionizing how businesses interact with data and users. This post explores the smooth integration of SAS Intelligent Decisioning with Azure OpenAI, showcasing how these technologies can transform AI assistants into powerful tools for real-time, data-driven decision-making.   Demonstrating the Concept: AI-Powered Risk Assessment   Imagine interacting with an AI assistant that not only answers your queries but also performs complex tasks, such as calculating your financial risk rating. Through a conversational interface powered by Azure OpenAI, the assistant collects inputs like age, income, employment type, debt-to-income ratio, and credit score. These inputs are then passed to a trusted internal workflow. The decision logic originates in SAS Intelligent Decisioning, which evaluates the data and returns an accurate risk rating (e.g., low, medium, or high).     This approach combines the ease of conversational AI with the precision of decision logic, providing an intuitive and reliable user experience. Whether it’s for banking, insurance, or other industries, this solution makes complex decision-making accessible and efficient.   Explaining the Concept: How It Works   The integration relies on a streamlined workflow involving key components:   Azure OpenAI Assistant   The assistant in Azure AI Foundry acts as the user interface, collecting inputs and triggering backend workflows. Using the Azure OpenAI Assistants API, the assistant can call external functions, such as an Azure Logic App, to process user requests.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   Azure Logic App Workflow   The Logic App serves as the bridge between the AI assistant and the decision logic. It:   Formats user inputs into a JSON payload. Sends the payload to an API endpoint hosting the decision logic. Processes the decision logic’s response and sends the result back to the assistant.       SAS Intelligent Decisioning   The decision logic resides in SAS Intelligent Decisioning, which evaluates the inputs and produces a result. For example, in a risk rating scenario, it calculates whether the risk is "low," "medium," or "high" based on predefined rules.       SAS Container Runtime (SCR) Containers: I decided to deploy the SAS decision logic to an Azure container and invoke it via the Logic App.   Deployment Options   The integration supports several deployment options:   SAS Viya SAS Container Runtime (SCR) Containers. SAS Decisioning Connector: Use the native connector in Logic Apps to call decisions published to SAS Viya Microanalytic Service. (The connector is currently in preview, and for many reasons, I did not manage to connect directly to a SAS Viya instance deployed in Azure. However, it's a solid option.)   Azure Instead of using the Azure AI Foundry interface, the approach can be packaged and deployed as a Python application. Further, the Python application can be containerized and deployed in Azure.   End-to-End Workflow: A Risk Rating Calculation Example   Here’s a step-by-step breakdown of the process:   User Interaction: The AI assistant offers to calculate a user’s risk rating. The assistant collects inputs such as age, income, and debt-to-income ratio. Data Transformation: The assistant converts these inputs into a JSON payload. API Invocation: The Azure Logic App sends the JSON payload to the SAS Intelligent Decisioning API. Decision Scoring: The decision logic processes the inputs and returns a risk rating. Response Delivery: The Logic App formats the result, and the assistant communicates it back to the user.   Conclusions: Transforming AI Assistants into Decision Engines   The integration of SAS Intelligent Decisioning with Azure OpenAI Assistants represents a significant leap in AI-driven workflows and an opportunity for SAS. By combining conversational AI with robust decision-making systems, businesses can deliver real-time, accurate insights while maintaining a uniform user experience.   Why This Matters:   Enhanced Efficiency: Automates decision-making processes. Improved Accuracy: Leverages trusted decision logic for reliable outcomes. User-Friendly: Simplifies complex workflows through a conversational interface.     By bridging conversational AI with decision logic, this integration delivers smarter, faster, and more intuitive workflows. Whether you’re automating risk assessments or other critical decisions, the combination of SAS and Azure technologies is a game-changer for AI-powered solutions.     Additional Resources   I was inspired by the following resources to power an Azure AI Assistant with a SAS decision:   Call Azure Logic Apps as Functions Using Azure OpenAI Assistants. Azure OpenAI Assistants API with Logic Apps. SAS Decisioning (Preview).   Thank you for your time reading this post. If you liked the post, give it a thumbs up! Please comment and tell us what you think about having conversations with your data. If you wish to get more information, please write me an email.     Find more articles from SAS Global Enablement and Learning here. ... View more

Imagine you’ve built a Python program that uses Generative AI (GenAI) to query data using natural language from any table in any SAS library in a SAS Viya environment. Now, you want to make it easier for users to consume this powerful application. The best approach? Containerization! This guide will walk you through containerizing the application so it’s portable, scalable, and easy to deploy.     The End Result   By the end of this process, you’ll have an interactive AI-powered assistant that enables users to query and analyze SAS Viya data using natural language. Powered by Azure OpenAI’s GPT-4 and SAS Viya, the application:   Converts user questions into SAS code. Executes the code in a SAS Viya Compute context. Summarizes the results in plain language.   This solution empowers users to focus on insights without needing programming skills.     Here's how to get started.     Container Ingredients   1. The AI Assistant   The application relies on a Large Language Model (LLM). For this guide, we use GPT-4 deployed on Azure OpenAI. The Python program communicates with the LLM through the OpenAI Python client library.     2. Environment Variables   To securely manage Azure OpenAI credentials, use environment variables. Create a .env file in the same directory as your Dockerfile with the following content:   OPENAI_URI=Your_Azure_OpenAI_Endpoint OPENAI_KEY=Your_Azure_OpenAI_API_Key OPENAI_VERSION=Your_Azure_OpenAI_API_Version OPENAI_GPT_DEPLOYMENT=Your_OpenAI_Model_Deployment_Name   Replace the placeholders with your actual Azure OpenAI details. This ensures sensitive information stays secure and outside your codebase. For Docker on Linux do not wrap the values in quotes.     3. The Python Program   The Python script (e.g., demo-function-calling-sas.py) is the core of the application. It orchestrates:   Dynamic SAS Viya URL Configuration: Allows users to work with any SAS Viya environment by dynamically updating the URL in the sascfg_personal.py file. Azure OpenAI Integration: Uses GPT-4 to interpret user queries and generate SAS code. SAS Code Execution: Executes the SAS code via the SASPy Python package and retrieves results. Interactive Workflow: Runs an interactive session where users can query data, switch tables, and receive insights.   import os import json from typing import Any, Callable, Dict from openai import AzureOpenAI from dotenv import load_dotenv from termcolor import colored import saspy import re # Load environment variables load_dotenv(".env") # Azure OpenAI configuration api_endpoint = os.getenv("OPENAI_URI") api_key = os.getenv("OPENAI_KEY") api_version = os.getenv("OPENAI_VERSION") api_deployment_name = os.getenv("OPENAI_GPT_DEPLOYMENT") # Create an AzureOpenAI client client = AzureOpenAI( api_key=api_key, api_version=api_version, azure_endpoint=api_endpoint) def sas_viya_url(): # Ask the user for the new URL new_url = input("Enter your SAS Viya URL, for example: https://beret-p04222-rg.gelenable.sas.com: ") # Read the entire content of the file with open('sascfg_personal.py', 'r') as f: content = f.read() # Define a regex pattern to find the 'url' key inside 'httpsviya' dictionary pattern = r"(httpsviya\s*=\s*\{\s*(?:[^{}]*\n)*?\s*'url'\s*:\s*)'[^']*'" # Replacement string with the new URL replacement = r"\1'{}'".format(new_url) # Perform the substitution new_content = re.sub(pattern, replacement, content, flags=re.MULTILINE) # Write the updated content back to the file with open('sascfg_personal.py', 'w') as f: f.write(new_content) print("The 'url' value has been updated successfully in sascfg_personal.py.") def log_message(message: str): """Logs error messages in red bold text.""" print(colored(message, 'red', attrs=['bold'])) def print_in_color(key, value): """Prints key-value pairs in color for better readability.""" if isinstance(value, str): formatted_value = value.replace("\\n", "\n").strip() else: formatted_value = str(value) # Convert to string if not already print(colored(key, 'green', attrs=['bold']), colored(formatted_value, 'yellow', attrs=['bold'])) def execute_sas_code(sas, sas_code: str) -> str: """Executes the given SAS code using the provided SAS session.""" sas_result = sas.submit(sas_code, results='TEXT') result_txt = sas_result['LST'] return result_txt def get_column_info(sas, library: str, table: str) -> str: """Retrieves column metadata from a SAS table.""" sas_code = f""" proc contents data={library}.{table} out=column_metadata(keep=name type length format informat label) noprint; run; proc print data=column_metadata; run; """ return execute_sas_code(sas, sas_code) # SAS Tools tools_list = [ { "type": "function", "function": { "name": "execute_sas_code", "description": "This function is used to answer user questions about SAS Viya data by executing PROC SQL queries, DATA STEP or any other SAS PROC against the data source.", "parameters": { "type": "object", "properties": { "sas_code": { "type": "string", "description": f""" The input should be a well-formed SAS code to extract information based on the user's question. The code execution result will be returned as plain text, not in JSON format. """, } }, "required": ["sas_code"], "additionalProperties": False, }, }, }, ] def call_functions(tool_calls, function_map) -> None: """ Processes and executes function calls requested by the assistant. This function is responsible for calling the appropriate Python functions based on the function calls detected in the assistant's response. It extracts the function name and arguments from each function call, looks up the corresponding function in `function_map`, executes it with the provided arguments, and displays the result. Args: tool_calls (list): A list of function call objects extracted from the assistant's response. Each function call contains the function name and its arguments. Raises: ValueError: If a function name from the tool call does not exist in `function_map`. """ for tool_call in tool_calls: func_name = tool_call.function.name arguments = json.loads(tool_call.function.arguments) print_in_color("Executing function tool call", "") print_in_color("Function Name:", func_name) print_in_color("Arguments:", arguments) function = function_map.get(func_name) if not function: raise ValueError(f"Unknown function: {func_name}") result_df = function(arguments) print(result_df) def process_message(sas, question: str, library: str, table: str, sas_table_info: str): """Processes the user's question and interacts with the OpenAI API. Calls functions as needed.""" # Construct system message system_message = ( "You are a data analysis assistant for data in SAS tables and libraries. " "Please be polite, professional, helpful, and friendly. " "Use the `execute_sas_code` function to execute SAS data queries, PROC SQL, DATA STEP or any other SAS PROC." "Default to aggregated data unless a detailed breakdown is requested. " "The function returns TXT-formatted results, nicely tabulated for use display. " f"Refer to the {library}.{table} metadata: {sas_table_info}. ", f"If a question is not related to {library}.{table} data or you cannot answer the question, " "then simply say, 'I can't answer that question. Please contact IT for more assistance.' " "If the user asks for help or says 'help', provide a list of sample questions that you can answer." ) #print(system_message) messages = [{"role": "system", "content": str(system_message)}, {"role": "user", "content": question}] # First API call: Ask the model to use the tool try: response = client.chat.completions.create( model=api_deployment_name, messages=messages, tools=tools_list, temperature=0.2, max_tokens=1512, ) # Process the model's response response_message = response.choices[0].message tool_calls = getattr(response_message, "tool_calls", []) # addition messages.append(response_message) #print("Model's response: ", response_message) # Map the tool to the function defined above function_map: Dict[str, Callable[[Any], str]] = { "execute_sas_code": lambda args: execute_sas_code(sas, args["sas_code"]), } if tool_calls: call_functions(tool_calls, function_map) # new addition for tool_call in response_message.tool_calls: if tool_call.function.name == "execute_sas_code": function_args = json.loads(tool_call.function.arguments) #print(f"Function arguments: {function_args}") sas_response = execute_sas_code(sas, sas_code=function_args.get("sas_code") ) messages.append({ "tool_call_id": tool_call.id, "role": "tool", "name": "get_current_time", "content": sas_response, }) else: return response_message.content # Second API call: Get the final response from the model final_response = client.chat.completions.create( model=api_deployment_name, messages=messages, temperature=0.2, max_tokens=888 ) return final_response.choices[0].message.content except Exception as e: log_message(f"An error occurred: {e}") # Main function def main(): """Main function to start the assistant.""" # Choose the SAS Viya URL for SASPY config sas_viya_url() # Start SAS session print ('I am starting a SAS session. Thanks for your patience...') sas = saspy.SASsession(cfgfile='sascfg_personal.py', cfgname='httpsviya') library = 'sampsio' table = 'dmlcens' print(f"Library and table selected: {library} and {table}") # Fetch table metadata once at the beginning sas_table_info = get_column_info(sas, library=library, table=table) print('I am going to use this table metadata: ', sas_table_info) while True: print("SAS Viya and Azure OpenAI are listening. Write 'help' to get suggestions, 'change' to switch tables, 'stop' or press Ctrl-Z to end.") try: # Start session q = input("Enter your question: \n") if q.strip().lower() == "change": library = input("Enter the library name: ") table = input("Enter the table name: ") print(f"Library and table selected: {library} and {table}") # Fetch new table metadata after changing the library/table sas_table_info = get_column_info(sas, library=library, table=table) print('I am going to use this table metadata: ', sas_table_info) elif q.strip().lower() == "stop": print("Conversation ended.") sas.endsas() break else: assistant_result = process_message(sas, q, library, table, sas_table_info) print(assistant_result) except EOFError: break except Exception as e: # Handle exceptions print(f"An error occurred: {e}") if __name__ == "__main__": main()   Key Features:   Metadata-Driven: Fetches table metadata (e.g., columns, types) to inform GPT-4’s responses. Custom Functionality: Exposes specific tools (e.g., execute_sas_code) to the LLM for executing SAS queries in SAS Viya using SASPy. Error Handling: Logs errors in a user-friendly format and handles exceptions gracefully.   Tip: A SAS Studio compute context session called sas is started just once, at the beginning of the main function. The session is reused in the def execute_sas_code(sas, sas_code: str) -> str: speeding up the application.     4. SAS Viya   The file sascfg_personal.py is needed to start a SAS Viya SAS Studio compute context session, using the SASPy Python package. The generated SAS code is executed in this session.   SAS_config_names=['httpsviya'] SAS_config_options = {'lock_down': False, 'verbose' : True, 'prompt' : True } SAS_output_options = {'output' : 'html5'} # not required unless changing any of the default httpsviya = {'url' : 'your-sas-viya-url.com', 'context' : 'SAS Studio compute context', 'options' : ["fullstimer", "memsize=4G"], 'verify' : False }   Tip: The Python program demo-function-calling-sas.py asks the user the SAS Viya environment URL and writes it in the sascfg_personal.py in the httpsviya object. The trick allows you to work with any SAS Viya environment. SASPY will then generate a link that you can use to log in SAS Viya and generate an authentication code.     5. Python Requirements File   The following Python packages are required. You can install them using pip and the provided requirements.txt file.   openai>=1.37.1, <2.0.0 python-dotenv>=1.0.1, <2.0.0 termcolor saspy     6. Docker   To build a Docker container you will need a machine with Docker installed. For this example, we’ll assume Docker is installed on a Linux machine (CentOS).     7. Dockerfile   The Dockerfile specifies how to build your Docker image, copying the necessary files, installing the requirements and running the Python program.   # Use a lightweight Python image FROM python:3.9-slim # Set the working directory in the container WORKDIR /app # Copy requirements.txt into the container COPY requirements.txt . # Install the Python dependencies RUN pip install --no-cache-dir -r requirements.txt # Copy the rest of the application code COPY . . # Set the entry point to your application CMD ["python", "demo-function-calling-sas.py"]     8 .Dockerignore File   A .Dockerignore file excludes sensitive files and unnecessary content from your Docker image.   .env __pycache__/ *.pyc .git/ /images     Build the Container Image   Then, use the Docker CLI to build your image from the Dockerfile:   docker build -t data-query-sas . # Confirm the build docker image ls | grep data-query-sas     Build the Container   Use the Docker CLI to create and run a container from the image you built:   docker run -it --rm --env-file .env data-query-sas python demo-function-calling-sas.py   Note that the container will use the environment variables stored in the .env file.     Try It Yourself   Why don’t you try the container yourself using the instructions and files in this post and let us know how it went?     Conclusion   A container and a few tricks allow you to transform a Python program in a fully working AI Assistant for SAS Viya.     Additional Resources   The approach is similar with the Jupyter Notebook described in Query Data Using Natural Language in SAS Viya with Azure OpenAI. The Python program has been tweaked to reuse a SAS Compute session, handle authorization code authentication with any SAS Viya environment and allow changing the table to be analyzed. The most important difference is that the program has been packaged as a Docker container.     Thank you for your time reading this post. If you liked the post, give it a thumbs up! Please comment and tell us what you think about having conversations with your data. If you wish to get more information, please write me an email.     Find more articles from SAS Global Enablement and Learning here. ... View more

Imagine you're working with a new dataset in SAS Viya but lack the time or skills to code. With Azure OpenAI Assistants function calling, you can ask questions directly from a Jupyter Notebook. The assistant translates your queries into SAS code, executes it, and delivers the results back in natural language. This interaction empowers you to explore data intuitively, focusing on insights rather than coding syntax.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   Data querying and data analysis using natural language on any dataset is now possible with Azure OpenAI's Assistant API, specifically through its Function Calling feature. This allows users to analyze data in SAS Viya without writing complex SAS code. By leveraging a simple Jupyter Notebook, users can interact with the assistant to ask questions in natural language. The assistant translates these questions into SAS code, executes it in the SAS Viya environment, and returns the results in a user-friendly format. SASPy, a Python package, facilitates the connection between Python and SAS Viya, enabling seamless data querying and analysis. This approach supports various data sources, including SAS libraries, databases, and cloud data sources, offering flexibility and efficiency. It empowers users to perform complex data analyses without technical skills, focusing on insights rather than programming.     Behind the Scenes   How does this happen? The process begins by connecting to a SAS Viya environment using SASPy, a Python package that facilitates interaction with SAS from Python. Before diving into data analysis, metadata about the dataset is gathered and incorporated into the Azure OpenAI Assistant's context. This metadata is crucial as it ensures the assistant generates precise and relevant SAS code without fabricating column names or data types.   The assistant's true power lies in its ability to convert natural language questions into SAS queries using the function execute_SAS_code. This function executes the generated SAS code within the SAS Viya environment, and the results are returned to the user. By injecting metadata early, the assistant is equipped to generate accurate queries, preventing errors and enhancing the overall reliability of the analysis process.     Conclusion   By leveraging the Azure OpenAI Assistants API and SAS Viya, data querying becomes accessible and efficient, even for those without technical expertise. This integration simplifies complex data queries into conversational interactions, democratizing data analysis and empowering a wider audience to make informed decisions. The future of data querying is here, and it's conversational, intuitive, and incredibly powerful.   Additional Resources   The approach is similar with the LangChain agents described in previous posts. The difference is that Microsoft offers now an alternative to LangChain through the Azure OpenAI Assistants API.   SWAT Code Generation and Execution in SAS Viya with Azure OpenAI and LangChain. SWAT Code Generation and Execution in SAS Viya with Azure OpenAI and LangChain: Behind the Scenes. GPT-4 Assisted Data Management in SAS Viya: A Custom LangChain Agent Approach. How to Create Your Custom LangChain Agent for SAS Viya.   Other posts:   How Accurate is GPT-4 at SAS Viya Data Management Tasks? How Retrieval Augmented Generation (RAG) Works? SASPY: Submit workloads to SAS Viya from Python. Conversing with Data: Turning Queries into Conversations with SAS Viya, Azure OpenAI and LangChain. Exploring LangChain and Azure OpenAI’s Ability to Write SQL and Join Tables To Answer Questions.   Thank you for your time reading this post. If you liked the post, give it a thumbs up! Please comment and tell us what you think about having conversations with your data. If you wish to get more information, please write me an email.     Find more articles from SAS Global Enablement and Learning here. ... View more

As the tech landscape evolves, so too do the tools and methods we use to secure our digital assets and workflows. If you’re working with Git repositories in SAS Studio on SAS Viya, you’ve likely noticed some changes in the Git authentication. Here’s a concise guide to help you navigate these updates and ensure a smooth transition.     What's Changing?   Since SAS Viya stable release 2024.06 (June 2024), when you access Git repositories in SAS Studio, you’ll encounter a warning message indicating that password-based authentication is being phased out. Clicking on the "More Details" hyperlink will direct you to the SAS documentation, which underscores the necessity of moving to an authentication domain for HTTPS authentication or SSH keys secured by a password.   By December 2024, this will be the only method available for Git repository authentication, so it's crucial to start the transition now.   Watch the following video that explains what is changing and how to adapt your Git profiles.       Cloning a Repository via HTTPS   Let’s walk through cloning a GitHub repository using the HTTPS method:   Copy the Repository URL: Obtain the URL from your GitHub repository. Specify a Server Location: This should be an empty folder on a network file system associated with SAS Viya. Create a New Git Profile: Choose the HTTPS option and provide the necessary details: Profile Name Commit Author Name (usually your user ID) Email associated with your user ID Credentials Method: Initially, you can enter a password, but by the end of 2024, you must use an authentication (password) domain. The password here is your GitHub Personal Access Token (PAT).     Storing Credentials in SAS Environment Manager   To store your PAT in My Credentials:   Navigate to SAS Environment Manager My Credentials. Create a New Credential: Associate it with an authentication domain (typically created by a SAS Administrator). Store your user ID and the PAT. The interface requires the user ID, although it plays no role in the Git authentication. We recommend you should match the user id in your Git HTTPS profile, inside SAS Studio, for consistency.     Using the Credential in SAS Studio   Once you store your credential in an authentication domain:   Go back to SAS Studio. Select your Git profile. Select the Authentication Domain associated with your password.   You can now clone the repository.      Bonuses   Obtaining a GitHub Personal Access Token To generate a PAT for GitHub:   Go to User Settings -> Developer Settings -> Personal Access Tokens. Select Tokens (Classic) and Generate a New Token. Choose Scopes: The minimal scopes are at least repo and user. Generate and Store the Token in your authentication domain in SAS Environment Manager.   Creating an Authentication Domain (for SAS Administrators) For SAS Administrators, here's how to create a new authentication domain:   Access SAS Environment Manager. Under Security > Domains > Create a New Authentication Domain named, for example, GitAuth. Set the Type to Authentication Password. Add a Description and Select Members: These can be individual SAS users or groups (e.g., a group called "Git Users"). Use the administrator credentials, to finalize the domain setup.   Now, Git users can store their credentials in this domain and use it in their Git profiles within SAS Studio.     Conclusion   The move to authentication domains for Git in SAS Studio on SAS Viya is a necessary step towards enhanced security. By following these steps, you can ensure a seamless transition and continue to manage your Git repositories efficiently. Don’t wait until the last minute—start integrating these changes into your workflow today.   Thank you for your time reading this post. If you liked the post, give it a thumbs up! Please comment and tell us what you think. If you wish to get more information, please write me an email.     Find more articles from SAS Global Enablement and Learning here. ... View more

A more streamlined title would be: "Comparing GPT-4 Models on SWAT Code Generation for SAS Viya: A Study of 18 Prompts." How accurate is GPT-4 at generating SWAT code to perform light data management tasks in SAS Viya? With eighteen sets of prompts we tested two custom agents: one using the “base” GPT-4 model versus a second using a GPT-4 model grounded in documents highly relevant for SWAT code generation. Which one performed better? Are there any significant advantages using a RAG approach? Read the post and watch the videos to find out more.    Left: DALL-E generated image. Right: "Show me Morpheus" image. Source: imgur.   In the context of the movie "The Matrix," Morpheus responded to a claim of knowledge, when Neo exclaimed "I know Kung Fu", with "Show me," indicating a request for a demonstration of the skills in question.   If a GPT-4 custom agent would state "I know SWAT" (SAS Wrapper for Analytics Transfer) as a parallel to Neo's line, my response would follow along Morpheus' lines, inviting the custom agent to demonstrate the knowledge in a relevant situation.   And here came the response to that hypothetical "Show me".   GPT-4 Base vs GPT-4 with RAG   In our experiments, we compared side-by-side the SWAT code generation skills of two Azure OpenAI's GPT-4 models, version 1106-preview:   The 'Base' model refers to the standard deployment of GPT-4. The 'GPT-4 with RAG' variant, on the other hand, was enhanced with a Retrieval-Augmented Generation process and informed by a collection of nineteen documents. These documents comprised posts by Peter Styliadis, focusing on Getting Started with Python Integration to SAS® Viya® which we downloaded and converted to Word files to serve as a knowledge base for the model.   This approach builds upon the methods we detailed in our previous post, SWAT Code Generation and Execution in SAS Viya with Azure OpenAI and LangChain: Behind the Scenes.   To understand the experiment, you might want to watch the following short video:     Summary of Results for Data Management Tasks Using GPT-4   After running eighteen distinct sets of prompts, we've compiled the outcomes of our experiments with two GPT-4 models: the standard 'Base' model and an enhanced version incorporating a Retrieval-Augmented Generation (RAG) technique. Here's how they performed:   Results  GPT-4 "Base" GPT-4 with RAG Successful 13 14 Partial Success (Different Results) 2 2 Unsuccessful 3 2 Total Tasks 18 18   Achieving a success rate of 15 or 16 out of 18 represents a strong performance. This suggests that both GPT-4 models are quite adept at handling light data management tasks, with the RAG-enhanced model showing a slight edge.   Nevertheless, we must approach these figures with a discerning eye. In the age of Business Intelligence (BI), it was not uncommon for five different dashboards to present five distinct sales figures. Language models, including the latest LLMs like GPT-4, haven't entirely resolved this issue. It's crucial to remember that while language models can significantly aid in data management tasks, the reliability of their outputs must be thoroughly vetted, particularly when those outputs inform critical business decisions.     Detailed Results for Data Management Tasks Using GPT-4   We prompted a series of data management tasks to evaluate the capabilities of two configurations of GPT-4, the 'Base' model and the enhanced 'GPT-4 with RAG' model. Our tasks varied in complexity:   Light Tasks: Listing caslibs, files, and tables. Medium Tasks: Generating table summaries, filters, top n results, group by operations, aggregations, and calculated columns. Heavy Tasks: Creating and saving tables, determining join columns for table joins, and promoting tables—some of the most challenging tasks for the model.   The models' performance should be viewed in light of their training data; the quality of their output is influenced by the data they've been exposed to during training. Here's how they fared:   ID Prompt GPT-4 "Base" GPT-4 with RAG Conclusion 1 List files Pass Pass Similar 2 List caslibs Pass Pass Similar 3 List in-memory tables Pass Pass Similar 4 Load a CSV from an URL to a promoted table Pass Pass Similar 5 Confirm the table has been loaded Pass Pass Similar 6 Column info Pass Pass (with an extra prompt). Base model slightly ahead. 7 Table summary statistics Pass Pass Similar 8 Describe a table Pass Pass Similar 9 Filter a table. Provide row counts. Pass Pass Different results. Trust issue. 10 New calculated column Pass Pass Similar 11 Top n Pass Pass Similar 12 Group by + aggregate. Pass Pass Different results. Trust issue. 13 Rename a column. Column info to confirm. Pass Pass Similar, RAG unaware of success. 14 Unique count for values in a column. Pass Pass RAG has better intent understanding. 15 Count the number of missing values from a table. Pass Pass RAG performs better, Base needs guidance. 16 Create a new promoted (global) table with a few lines of data. Fail Pass RAG handles promotion well. Base model fails to promote. 17 Join a table with the newly created table. Model must figure the key to join on. Fail Fail Task is challenging for both models. 18 Save a table and promote it. Filter an existing table, save as a promoted table. Fail Fail Both models struggle with table saving.   Ten Rounds of Prompts   This video presents a head-to-head challenge of ten rounds, where we prompt the model to tackle a series of data management tasks in SAS Viya. The tasks range from simpler ones, such as describing columns and summarizing tables, to more complex operations like creating calculated columns, sorting, and identifying top values. We also cover grouping with aggregations, renaming columns, performing unique counts, and saving tables after applying filters.   In most of the cases, both agents succeed at the given tasks, with the same results. Sometimes they both succeed but the results are different! Sometimes the RAG agent needs an extra “nudge” or further instructions. At other times, the RAG agent succeeds where the other failed or they both fail.   I won’t comment the full 28 minutes of the video, although I added a few explanations. Enjoy watching or scrolling through!     Excellence or Instead of Conclusions   Two custom agents were tested:   GPT-4 "Base" excels in 2 tasks: column info and rename a column followed by column info. GPT-4 with RAG shows superiority in 3 tasks: unique counts, missing values, promoting tables.   Overall, the results slightly favor the GPT-4 with RAG model, indicating a marginal edge in understanding and executing complex data management tasks.   Ultimately, the performance difference between the two models is relatively small. Considering the additional resources and time required to set up the RAG, one must weigh these against the need for precision.   For rapid outcomes where the highest accuracy is not critical, the 'Base' model is your go-to option. It provides quick results without the extra setup. The GPT-4 model 1106-preview version is a far cry compared with an earlier davinci-code-003 model I tested for SAS code generation.   However, if your priority is tailored accuracy and you're dealing with complex tasks where nuanced understanding is key, the 'GPT-4 with RAG' model is likely the better choice, despite the additional investment.   The study emphasizes the importance of verifying the output of language models, especially when informing critical business decisions.   I hope you found this article insightful. Please feel free to reach out with feedback or suggestions for enhancing the agent or taking its capabilities to the next level.     Acknowledgements   Thanks to Peter Styliadis for his great SWAT Series.   Additional Resources   How Retrieval Augmented Generation (RAG) Works? SWAT Code Generation and Execution in SAS Viya with Azure OpenAI and LangChain: Behind the Scenes. SWAT Code Generation and Execution in SAS Viya with Azure OpenAI and LangChain. SASPY: Submit workloads to SAS Viya from Python. LangChain Custom Agent. GPT-4 Assisted Data Management in SAS Viya: A Custom LangChain Agent Approach. How to Create Your Custom LangChain Agent for SAS Viya. Conversing with Data: Turning Queries into Conversations with SAS Viya, Azure OpenAI and LangChain. Exploring LangChain and Azure OpenAI’s Ability to Write SQL and Join Tables To Answer Questions.   Thank you for your time reading this post. If you liked the post, give it a thumbs up! Please comment and tell us what you think about the approach. If you wish to get more information, please write me an email.     Find more articles from SAS Global Enablement and Learning here. ... View more

Retrieval Augmented Generation (RAG) is an architecture that augments the capabilities of a Large Language Model (LLM) like ChatGPT by adding an information retrieval system that provides grounding data. Let’s start with some Large Language Model (LLM) generated poetry:   To RAG or not to RAG, that is the query, A question of depth, not at all imaginary. Shall we enhance our LLM with files galore, Or trust its vast knowledge, needing no more?   To RAG or not to RAG that is the question we would like to answer in this post.   RAG Data Sources   Azure OpenAI facilitates RAG by integrating pretrained models with your unique datasets. You can augment the prompts sent to the model by incorporating a data source of your choice. This can be done by uploading your files directly, utilizing existing blob storage, or connecting to a pre-existing AI Search index. Azure OpenAI supports a range of file types, including .md, .txt, .html, .pdf, as well as .docx and .pptx documents. It's important to note that if these files include graphics or images, the quality of the model's response will hinge on the effectiveness of text extraction from these visuals.   Index   For indexing, we utilize Azure AI Search, which is distinct from Azure OpenAI. This component is designed to construct a search index from your data.   Search   Azure OpenAI leverages Azure AI Search to enhance prompts by appending pertinent data snippets. By default, the system prompts the model to prioritize, but not exclusively rely on your data. However, this preference can be adjusted during setup, allowing the model to balance its pre-trained knowledge with the information from your data sources.   Retrieval Augmented Generation Overview Architecture Diagram   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   How to Implement Retrieval Augmented Generation (RAG) with Azure OpenAI   Watch the following video that explains how to add a data source to your model. The video compares the code generation question answers given by the "base" GPT-4 model versus the GPT-4 model grounded in your data.     As an intermediary conclusion, RAG could be an answer when the code you are trying to generate is "exotic" or proprietary or not covered by many public sources.   A Second RAG Example Loading a SAS Data Set into CAS – A Comparison   Suppose you are prompting GPT-4 ‘I'd like to perform an efficient client-side SAS data set load in memory, in CAS. Let's assume the data set is PRDSALE from SASHELP. I want to load it in CASUSER caslib.’   Without RAG: The code generation process would rely solely on pre-defined scripts and the programmer's knowledge to accomplish the task.   With RAG: The code generation is enhanced by dynamically pulling in relevant information from an index, potentially improving efficiency and accuracy of the generated code snippet for loading the data set into CAS.     CAS Code Generation with RAG   Let's explore how RAG streamlines the process of generating CAS code using a single markdown file from a GEL Data Management workshop that outlines efficient client-side in-memory data loading:   Once the file is uploaded to an Azure Storage Account and ingested by Azure AI Search to create an index, Azure OpenAI is ready to tap into this data source.     Here's what happens when you request SAS code to load a data set into CAS:   Azure OpenAI receives the user prompt, which in this case is a request to load the PRDSALE data set from SASHELP into the CASUSER caslib. It analyzes the prompt to grasp the desired content and intent, identifying the indexed file as highly relevant. The model then queries the search index using this intent. It incorporates the search results into the prompt, enriching it with both the system message and the user's original request. This enhanced prompt is sent to Azure OpenAI for processing. Azure OpenAI generates the SAS code, referencing the data where applicable, and returns it to the user.   And that's a brief rundown of RAG in action for CAS code generation.   Instead of a Conclusion   To RAG or not to RAG, ponder we must, For each choice we make bears its own kind of trust. Do we value the breadth, the external clout, Or the depth of the mind, what it’s all about?   Stay Tuned   In a next post, we will compare a SWAT custom LangChain agent powered by a GPT-4 model with a GPT-4 model + RAG on documents highly relevant for SWAT code generation. The comparison is based on eighteen prompts, asking the model to perform light, medium and increasingly difficult data management tasks in SAS Viya.   Additional Resources     Retrieval Augmented Generation (RAG) in Azure AI Search. SWAT Code Generation and Execution in SAS Viya with Azure OpenAI and LangChain. SWAT Code Generation and Execution in SAS Viya with Azure OpenAI and LangChain: Behind the Scenes. LangChain Custom Agent. GPT-4 Assisted Data Management in SAS Viya: A Custom LangChain Agent Approach. How to Create Your Custom LangChain Agent for SAS Viya. Conversing with Data: Turning Queries into Conversations with SAS Viya, Azure OpenAI and LangChain. Exploring LangChain and Azure OpenAI’s Ability to Write SQL and Join Tables To Answer Questions.     Thank you for your time reading this post. If you liked the post, give it a thumbs up! Please comment and tell us what you think about having conversations with your data. If you wish to get more information, please write me an email.     Find more articles from SAS Global Enablement and Learning here. ... View more

The SAS Scripting Wrapper for Analytics Transfer (SWAT) custom agent is designed to be an interactive AI assistant. It streamlines SAS code generation and execution for users, regardless of their expertise in SAS, Python, or CAS actions. In this post, we'll look into the agent's code, outline the prerequisites, and guide you through replicating the agent in your own environment.   Previously   Check out the video in SWAT Code Generation and Execution in SAS Viya with Azure OpenAI and LangChain for an insider's look at how this agent functions. I'll walk you through its operations, potential uses, and some of the intriguing behaviors that the custom agent displays.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   Now it's time to reveal the code and walk you through the details.   Custom Agent Code   The core Python script, named sasgenexswat.py, establishes the SWAT LangChain custom agent that interfaces with SAS Viya   # LangChain custom SAS Viya agent with memory and only two tools # Resource: https://python.langchain.com/docs/modules/agents/how_to/custom_agent import os import json import urllib3 urllib3.disable_warnings # Load the Large Language Model - Azure OpenAI from langchain_openai import AzureChatOpenAI from openai import AzureOpenAI # Get Azure OpenAI environment variables from .env from dotenv import load_dotenv load_dotenv() azure_oai_endpoint = os.getenv("AZURE_OAI_ENDPOINT") azure_oai_key = os.getenv("AZURE_OAI_KEY") azure_oai_deployment = os.getenv("AZURE_OAI_DEPLOYMENT") azure_oai_model = os.getenv("AZURE_OAI_DEPLOYMENT") os.environ["OPENAI_API_TYPE"] = "azure" os.environ["AZURE_OPENAI_API_KEY"] = azure_oai_key os.environ["AZURE_OPENAI_ENDPOINT"] = azure_oai_endpoint os.environ["OPENAI_API_VERSION"] = "2024-05-01-preview" ## We'll use GPT-4 as deployed model llm = AzureChatOpenAI(deployment_name=azure_oai_model, temperature=0, max_tokens=2000) print("We'll be using the following LLM:\n", llm, "\n") # LLM Client from openai import AzureOpenAI client = AzureOpenAI( azure_endpoint = azure_oai_endpoint, api_key = azure_oai_key, api_version= "2024-05-01-preview" ) # Arguments import sys # Check if at least one argument is provided if len(sys.argv) > 1: # sys.argv[1] is the first parameter passed to the script sas_viya_url = sys.argv[1] print(f"""Passing prompts to Azure OpenAI. \nAzure OpenAI will generate the SWAT code and execute it on: {sas_viya_url}. \nAssuming you obtained an access token first by running get_token_2023.py.""") print('\nRetrieving the saved token from api/access_token.txt') with open("api/access_token.txt", "r", encoding="UTF-8") as f: token = f.read() else: print("No SAS Viya URL provided.") import swat # SWAT session # Define custom tools - SWAT GenEx from langchain.pydantic_v1 import BaseModel, Field from langchain.tools import BaseTool, StructuredTool, tool import re # code parsing from io import StringIO # exec output @tool def get_word_length(word: str) -> int: """Returns the length of a word.""" return len(word) @tool def generate_swat_code(prompt: str) -> str: """Returns the generated SWAT Python code, based on the user prompt. The code should be aimed at printing the execution output, as it would be used further. Args: prompt: The prompt which will be sent to Azure OpenAI. """ response = client.chat.completions.create( model=azure_oai_model, messages=[ {"role": "system", "content": "You are a SAS Python SWAT package assistant that helps people write SWAT code. Return only the Python SWAT code, ready to be executed. This is the code in between ```python and ```. Assume there is already a connection named 'conn' defined."}, # ... other system and user messages ... {"role": "user", "content": prompt}, ] ) sas_response = response.choices[0].message.content + '\n' # Strip code of characters pattern = re.compile(r"py\n(.*?)", re.DOTALL) match = pattern.search(sas_response) if match: swat_code = match.group(1).strip() print('\n',swat_code, '\n') else: swat_code = sas_response # Clean code block code_block = swat_code swat_code = code_block.replace("```python\n", "").replace("\n```", "") # Save the SWAT code to a file filename = "generated_swat_code.py" # Open the file first try: with open(filename, 'r', encoding='utf-8') as file: current_content = file.read() except FileNotFoundError: current_content = "" # Prepend the new SWAT code to the existing content new_content = '# Prompt -> ' + prompt + '\n' '# Generated code -> ' + '\n' + swat_code + '\n' + current_content with open(filename, 'w', encoding='utf-8') as file: file.write(new_content) print(f"Generated SWAT code has been saved to {filename}") return swat_code @tool def execute_swat_code(swat_code: str) -> str: """Useful when you need to execute generated Python SWAT code. Input to this tool is correct Python SWAT code, ready to be executed in a SAS Viya environment CAS session. Assume an already established SWAT connection, called 'conn'. Identify the results, summarize and provide a short explanation. If an error is returned, you may need to rewrite the SWAT code, using the generate_SWAT_code tool. Provide a Final Result: Summarize and provide a short explanation of what was done and what the outcome was. If you encounter an issue, detail what the issue is. Args: swat_code: The SWAT Python code which will be submitted to a CAS session in SAS Viya. """ # Python code as a string stored in a variable print (f'\nI will submit the following code to {sas_viya_url}: \n',swat_code,'\n...End of code...\n') try: # Create SWAT connection using the url and a token obtained earlier port='443' conn_string= sas_viya_url + ':' + port + '/cas-shared-default-http' conn=swat.CAS(conn_string, password=token) # Create a dictionary to hold the execution context namespace = {'conn': conn} # Add 'conn' to the namespace old_stdout = sys.stdout old_stderr = sys.stderr redirected_output = sys.stdout = StringIO() redirected_error = sys.stderr = StringIO() try: # execute the SWAT code exec(swat_code, namespace) except Exception: # Optionally, handle the exception in some way here pass # Reset stdout and stderr to their original values sys.stdout = old_stdout sys.stderr = old_stderr # Get the captured output and errors output = redirected_output.getvalue() errors = redirected_error.getvalue() # Print the captured output and errors print("Captured Output:") print(output) print("Captured Errors:") print(errors) # Terminate SWAT connection conn.terminate() return output + errors except Exception as e: print(f'An error occurred: {e}') errors = e return errors # Describe the tools. The description is important for the LangChain chain. tools = [get_word_length, generate_swat_code, execute_swat_code] # LangChain Agent with Memory from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder MEMORY_KEY = "chat_history" prompt = ChatPromptTemplate.from_messages( [ ( "system", "You are a world class AI assistant who helps people generate and execute SAS code. Interpret the results if asked.", ), MessagesPlaceholder(variable_name=MEMORY_KEY), ("user", "{input}"), MessagesPlaceholder(variable_name="agent_scratchpad"), ] ) llm_with_tools = llm.bind_tools(tools) from langchain_core.messages import AIMessage, HumanMessage from langchain.agents.format_scratchpad.openai_tools import ( format_to_openai_tool_messages, ) from langchain.agents.output_parsers.openai_tools import OpenAIToolsAgentOutputParser chat_history = [] agent = ( { "input": lambda x: x["input"], "agent_scratchpad": lambda x: format_to_openai_tool_messages( x["intermediate_steps"] ), "chat_history": lambda x: x["chat_history"], } | prompt | llm_with_tools | OpenAIToolsAgentOutputParser() ) from langchain.agents import AgentExecutor agent_executor = AgentExecutor(agent=agent, tools=tools, verbose=True) # Main function def chat_with_open_ai(): # ... as before while True: print("SAS and Azure OpenAI are listening. Write 'Stop' or press Ctrl-Z to end the conversation.") try: # SWAT start session #conn = start_or_retrieve_swat_session(sas_viya_url) # Get text from the keyboard q = input("Enter your question: \n") if q == "Stop": print("Conversation ended.") #term = conn.terminate() #print('SWAT session ended successfully.', term) break else: print("\nI will ask: ", q) result = agent_executor.invoke({"input": q, "chat_history": chat_history}) chat_history.extend( [ HumanMessage(content=q), AIMessage(content=result["output"]), ] ) except EOFError: break # main try: chat_with_open_ai() except Exception as err: print("Encountered exception. {}".format(err))   Custom Agent Code Explanation   The provided Python code integrates several components to create a custom agent that leverages Azure's OpenAI Large Language Model (LLM), specifically GPT-4, to generate and execute SAS code via the SAS Python SWAT (SAS Scripting Wrapper for Analytics Transfer) package. This agent is designed to facilitate interactions with SAS Viya, a cloud-enabled, in-memory analytics engine.   Here's a step-by-step breakdown of the code's functionality:   Imports and Environment Setup: The script starts by importing necessary Python modules. It loads environment variables from a .env file, which includes credentials and endpoints for Azure OpenAI services. The AzureChatOpenAI class from the langchain_openai package is instantiated with parameters set for the LLM, such as the model's deployment name and the maximum number of tokens to generate. Argument Handling: The script checks for command-line arguments, expecting at least one argument: the SAS Viya URL. If provided, it reads an access token from a local file to use for authentication with the SAS Viya server. Custom Tools Definition: Several functions are decorated with @tool from the langchain package, indicating that they are custom tools that can be used within the agent's execution flow. generate_swat_code tool sends the user prompt to Azure OpenAI, which returns the generated SWAT Python code. Letting an LLM to generate the code is the innovative aspect of the custom agent. @tool def generate_swat_code(prompt: str) -> str: """Returns the generated SWAT Python code, based on the user prompt. The code should be aimed at printing the execution output, as it would be used further. Args: prompt: The prompt which will be sent to Azure OpenAI. """ response = client.chat.completions.create( model=azure_oai_model, messages=[ {"role": "system", "content": """You are a SAS Python SWAT package assistant that helps people write SWAT code. Return only the Python SWAT code, ready to be executed. This is the code in between ```python and ```. Assume there is already a connection named 'conn' defined."}, # ... other system and user messages ... {"role": "user", "content": prompt}, ] ) The code is then cleaned and saved to a file. execute_swat_code retrieves the code from the generate_swat_code, establishes a SWAT connection, and then tries to run the generated SWAT code on a SAS Viya server using the SWAT package. It captures the output and errors, returning them to the agent. The SWAT connection is then terminated to liberate CAS compute resources on the SAS Viya server. @tool def execute_swat_code(swat_code: str) -> str: """Useful when you need to execute generated Python SWAT code. Input to this tool is correct Python SWAT code, ready to be executed in a SAS Viya environment CAS session. Assume an already established SWAT connection, called 'conn'. Identify the results, summarize and provide a short explanation. If an error is returned, you may need to rewrite the SWAT code, using the generate_SWAT_code tool. Provide a Final Result: Summarize and provide a short explanation of what was done and what the outcome was. If you encounter an issue, detail what the issue is. Args: swat_code: The SWAT Python code which will be submitted to a CAS session in SAS Viya. """ try: # Create SWAT connection using the url and a token obtained earlier port='443' conn_string= sas_viya_url + ':' + port + '/cas-shared-default-http' conn=swat.CAS(conn_string, password=token) # Create a dictionary to hold the execution context namespace = {'conn': conn} # Add 'conn' to the namespace old_stdout = sys.stdout old_stderr = sys.stderr redirected_output = sys.stdout = StringIO() redirected_error = sys.stderr = StringIO() try: # execute the SWAT code exec(swat_code, namespace) except Exception: # Optionally, handle the exception in some way here pass # The SWAT code runs inside the exec(swat_code, namespace) function, after establishing a connection within the namespace. However, using exec means entrusting code execution to a custom agent without scrutiny, which isn't the most secure method for submitting SWAT code to SAS Viya. If you're aware of safer alternatives, I'm eager to hear them. LangChain Agent with Memory: A chat prompt template is defined using the LangChain package, which describes the behavior of an AI assistant capable of interpreting and acting on user input. The llm_with_tools object binds the custom tools to the LLM. from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder MEMORY_KEY = "chat_history" prompt = ChatPromptTemplate.from_messages( [ ( "system", "You are a world class AI assistant who helps people generate and execute SAS code. Interpret the results if asked.", ), MessagesPlaceholder(variable_name=MEMORY_KEY), ("user", "{input}"), MessagesPlaceholder(variable_name="agent_scratchpad"), ] ) llm_with_tools = llm.bind_tools(tools) The agent object is created by composing several functions and components, including the prompt, LLM, and an output parser. An AgentExecutor is instantiated, which handles the execution of the agent with the custom tools. Main Function (chat_with_open_ai): The main loop of the function listens for user input and uses the agent_executor to process the input, generating and executing SAS code accordingly. The chat history is updated with each interaction. The user can type "Stop" to end the conversation. The script is wrapped in a try-except block to handle any exceptions that occur during the chat loop. Execution: Finally, the chat_with_open_ai function is called to start the interactive session.       Now let’s look at what you need to build this custom agent.   SAS Viya Environment   To build this custom agent, you'll require a SAS Viya deployment. From your SAS Viya environment, you'll need to gather the following details: The SAS Viya URL A SAS username The corresponding SAS user's password   Additionally, ensure that CAS (Cloud Analytic Services) access is enabled for external connections to SAS Viya.   Azure Deployment If your SAS Viya is hosted on Azure, it's essential to configure an inbound security rule. This rule must include the IP address of the computer running Visual Studio Code and specify port 443. The target for this rule should be the public IP address of your SAS Viya's load balancer.   Azure OpenAI   To get started, you'll need to create a resource and deploy a model. If you're unfamiliar with the process, the Azure OpenAI QuickStart provides step-by-step guidance. To interact with Azure OpenAI, you require an endpoint, a key, and a deployment.   For this project, I selected the 1106-Preview GPT-4 model version, which was trained on data from December 2023. I've found that this version excels at generating SWAT code. Remember, choosing the right model is as crucial as the code that powers your application   Visual Studio Code   We will use Visual Studio Code with the Python extension installed to run the custom agent. If you’re a regular user of Visual Studio Code, great.   Python Requirements   Create a requirements.txt file that includes these Python packages: python-dotenv pandas urllib3 swat openai langchain langchain_openai langchain-core langchain-community   Azure OpenAI Configuration In the directory containing your main Python script, create a .env file to store your Azure OpenAI configuration parameters. The file should include the following lines: AZURE_OAI_ENDPOINT='https://myuser.openai.azure.com/' AZURE_OAI_KEY='key_here' #Azure Open AI key here AZURE_OAI_DEPLOYMENT='gpt-4' The sasgenexswat.py program is configured to read these environment variables from your .env file.   SWAT Setup Certificates Firstly, you'll need to obtain the SAS Viya certificates, which are in the .PEM file format, and download them to your local machine.   For detailed instructions, please refer to the provided documentation.   Get a SAS Viya Access Token   The sasgenexswat.py program is designed to read an access token from a file called api/access_token.txt.   For SWAT to establish a connection within the custom agent, you'll need an OAuth token. You have two options for obtaining this token:   Use any method you're comfortable with to acquire an access token and save it to api/access_token.txt. Follow the instructions in the post Discover Your Data with SAS Information Catalog APIs from Python – Access to run a program called get_token_2023.py. The program requires several variables.   Variables Once you have your certificates, open Visual Studio Code, launch a Bash terminal, and proceed with the setup. SASVIYAURL=my_sas_viya_url SASUSER=fill_in_here # SAS user SASPASS=pass_here # SAS user password CLIENT=fill_in_here # client id CLIENTSECRET=fill_in_here # client secret PEMPATH='fill_here' # for example PEMPATH='C:\Users\myuser\Downloads\trustedcerts.pem' export CAS_CLIENT_SSL_CA_LIST=${PEMPATH}   Get a Token Run a program called get_token_2023.py. You'll need to provide several environment variables. python get_token_2023.py $SASVIYAURL $CLIENT $CLIENTSECRET $SASUSER $SASPASS $PEMPATH   Install Python Packages   To install the Python packages listed in the requirements.txt file, execute the following command: pip install -U -r requirements.txt For additional context, if you're interested in the specific versions of LangChain, OpenAI, and SWAT packages used in the demo, you can retrieve them using these commands: pip freeze | grep lang pip freeze | grep openai pip freeze | grep swat python --version At the time of writing this post, the versions of the Python packages I am using are as follows: langchain==0.2.14 langchain-community==0.2.12 langchain-core==0.2.33 langchain-openai==0.1.22 langchain-text-splitters==0.2.2 langsmith==0.1.93 langchain-openai==0.1.22 openai==1.41.0 swat==1.14.0 Python 3.11.9   Run the Program   To execute the program, you'll need to provide your SAS Viya URL, which you've previously set as an environmental variable. Run the program by typing the following command: python sasgenexswat.py $SASVIYAURL   Prompts   Check out the video in SWAT Code Generation and Execution in SAS Viya with Azure OpenAI and LangChain for prompts and responses.   Start asking questions such as:   List all files in the SAMPLES caslib List all CASLIBS. List all available in-memory tables in the 'Public' caslib. Load a CSV file from a URL into the Public caslib and promote the table. URL is  'https://support.sas.com/documentation/onlinedoc/viya/exampledatasets/cars.csv'   List all available in-memory tables in the 'Public' caslib. Confirm CARS has been loaded. Print the columns of the CARS table. Request simple summary statistics for the CARS table from Public caslib. Describe the table CARS in the Public caslib. How many rows in the CARS CAS table from Public have the Make Toyota or Acura? From CARS, create a new calculated column NewCol2 = round(((MSRP - Invoice) / Invoice) * 100 and print three records to confirm the column creation. Find the top 10 MPG_City consumption values ordered ascending. Print the car make and model. Group the CAS table CARS from Public by Origin. Aggregate by column Invoice. Use means as a measure. Print the aggregation. Stop or CTRL+Z when you want to exit the program.   Conclusions   The SWAT code Generator-Executor (GenEx) custom agent approach is innovative, as it uses a large language model, GPT-4, to generate Python SWAT code based on user prompts. The custom agent is relying on just two main components: a SWAT code generator and a SWAT code executor. These tools work in tandem to run the generated SAS code within your chosen SAS Viya environment, managing both the output and any potential errors through the agent, which then crafts a final response.   SWAT code execution is the crucial step that makes large language model (LLM) code generation practical for SAS Viya users; it determines whether the code functions correctly in their environment.   Should an error arise, the agent is programmed to interpret the issue and attempt to regenerate the code, aiming for a successful execution. This can be a double-edged sword; while it's a source of strength, it can also be a weakness, as the custom agent may occasionally become stuck in a loop.   It's remarkable how just two hundred fifty lines of code can integrate enterprise analytics and data management platforms such as SAS Viya with AI platforms like Azure OpenAI. The linchpin that unites these diverse platforms remains open source.   I hope you found this article insightful. I encourage you to give it a try! Please feel free to reach out with feedback or suggestions for enhancing the agent or taking its capabilities to the next level.     Acknowledgements   Special thanks to Grace Liao from SAS Research & Development for inspiring me to try the approach.   Thanks to Peter Styliadis for his great SWAT Series.   Additional Resources   SWAT Code Generation and Execution in SAS Viya with Azure OpenAI and LangChain. LangChain Custom Agent. GPT-4 Assisted Data Management in SAS Viya: A Custom LangChain Agent Approach. How to Create Your Custom LangChain Agent for SAS Viya. Conversing with Data: Turning Queries into Conversations with SAS Viya, Azure OpenAI and LangChain. Exploring LangChain and Azure OpenAI’s Ability to Write SQL and Join Tables To Answer Questions.    Thank you for your time reading this post. If you liked the post, give it a thumbs up! Please comment and tell us what you think about having conversations with your data. If you wish to get more information, please write me an email.     Find more articles from SAS Global Enablement and Learning here. ... View more

The purpose of this custom agent is to create an interactive, AI-powered assistant that can simplify the process of generating and executing SAS code for users who may not be experts in SAS, Python or may not know anything about CAS and CAS actions. This can be particularly useful for data analysts and scientists who want to leverage the power of SAS Viya for complex data processing and analytics tasks without having to delve deeply into SWAT or SAS programming. The SWAT (SAS Scripting Wrapper for Analytics Transfer) package is a Python interface to SAS Cloud Analytic Services (CAS).   By leveraging the natural language understanding capabilities of Azure's OpenAI large language model (LLM), users can interact with SAS Viya in a conversational manner, asking questions or requesting actions to be taken on their data in SAS Viya. This custom agent serves as an interface that abstracts the complexity of coding by generating the necessary SWAT code from user prompts and executing it, thus allowing users to focus more on the analytical aspects of their work rather than the technical details of code syntax and structure.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   The Big Picture   This custom agent integrates multiple technologies, including SAS Viya, Azure OpenAI, and open-source Python packages like SWAT and LangChain.   The diagram depicts two circuits: the blue circuit and the red circuit.   In the blue circuit:   An LLM is at the core of LangChain custom agent. I am using GPT-4, from Azure OpenAI. The Generator tool inside the LangChain custom agent leverages a large language model (LLM), GPT-4, from Azure OpenAI, to generate SWAT code, basically Python, based on user prompts. I specifically chose a 1106-Preview model version. The 1106-Preview was trained on Dec 2023 data. I noticed this version "understands SAS better" and is naturally proficient at generating Python code, including SWAT. Choosing your model is as important as the programming behind the scenes. The Executor tool receives the generated SWAT code, creates a SWAT connection and is instructed to execute the generated Python code, the close the SWAT connection.   In the red circuit:   SWAT code execution logs and results are parsed and passed to the LangChain custom agent. At this point the LLM decides if the execution is successful. If so, it formulates a final result and prints it on the screen. If the execution results in errors or could not happen for any reason, a feedback loop starts. The Generator tool receives new instructions and new SWAT code is generated, and the whole process continues, like in the "blue circuit".   I like this approach, because of its simplicity. The custom agent requires only two custom tools, the generator and the executor.     Video   Watch the following video, where we explore the approach:   Possible Applications   The applications of the custom SWAT agent cover could enhance the user experience with SAS Viya. Here's a possible list of applications:   Automated Code Generation: This feature significantly streamlines the process of writing SWAT code. It's especially beneficial for users who may not be well-versed in SAS syntax, as it allows them to interact with SAS Viya through simple English prompts, making the analytics platform more user-friendly.   Power Under the Hood: This agent taps into the robust capabilities of the CAS engine for efficient data management tasks, such as loading, managing, and saving data from various sources. It can also utilize predefined connections to data sources known as caslibs, maximizing the utilization of existing infrastructure.   Accessibility and Inclusivity: By simplifying the complex aspects of programming, the agent lowers the barrier to entry for advanced analytics tools. This democratization of data science ensures that a broader audience can access and leverage powerful analytics capabilities.   Self-Correcting Feedback Loop: The inclusion of a self-correcting feedback loop where the agent reviews execution results and, in the case of errors, restarts the generation and execution cycle, introduces a layer of resilience and efficiency to the analytics process.   Interactive Analytics: The agent's conversational interface fosters an engaging and real-time interaction with the SAS Viya platform. Users can ask questions and receive immediate feedback, which enhances the analytical experience and accelerates decision-making.   Memory and Plasticity   The agent displays “plasticity”. The term "plasticity" is often associated with flexibility and adaptability. Here, it's used to describe the remarkable ability of an intelligent agent to adjust its actions and remember previous instructions.   Imagine you're working with an intelligent agent, and you've given it two tasks. The first task (Task A) is to load a table into memory, and the second task (Task B) is to query that table once it's loaded. If the agent encounters a problem with Task A, you can instruct it to try an alternative approach. Once Task A is successfully completed, the agent doesn't forget about Task B. Instead, it recalls that Task B was the next step and proceeds to perform it. This retention and sequential execution of tasks showcases the agent's memory capabilities.   This form of adaptability stems from the design of the LangChain agent. In this architecture, a language model (LLM) acts as a conduit between various tools and the custom behaviors you've programmed into those tools. In this scenario, I was clear about needing a code rewrite if the initial attempt failed. The agent's ability to handle this instruction is a proof to the flexibility built into its operation.   @tool def execute_swat_code(swat_code: str) -> str: """Useful when you need to execute generated Python SWAT code. Input to this tool is correct Python SWAT code, ready to be executed in a SAS Viya environment CAS session. Assume an already established SWAT connection, called 'conn'. Identify the results, summarize and provide a short explanation. If an error is returned, you may need to rewrite the SWAT code, using the generate_SWAT_code tool. Provide a Final Result: Summarize and provide a short explanation of what was done and what the outcome was. If you encounter an issue, detail what the issue is. Args: swat_code: The SWAT Python code which will be submitted to a CAS session in SAS Viya. """     Conclusions   By integrating the intuitive language understanding of Azure OpenAI's GPT-4 with the practical functionality of SWAT and LangChain, this agent removes the barrier of complex code, allowing users to command their simple data management tasks through simple conversation. The agent displays some "plasticity" – its ability to remember and adapt to tasks – when given a memory of previous conversations. This innovative approach may democratize data processing, empowering a broader range of professionals to make data-driven decisions swiftly and confidently.   I hope you enjoyed this article. And please contact me if you have any feedback or any ideas on how to improve the agent or take it one level further.   What's Next   Discover the inner workings of SWAT Code Generation and Execution in SAS Viya with Azure OpenAI and LangChain: Behind the Scenes. I'll unveil the custom agent Python program and detail the prerequisites for running it in your own environment.   In future posts, I will:   Add RAG to the custom agent. RAG or Retrieval-Augmented Generation, is a technique used in natural language processing (NLP) that combines the power of language models with the ability to retrieve information from relevant documents. Specifically for SWAT, I would be using Peter Styliadis' series and see if we achieve better results at SWAT code generation. Compare the "vanilla" custom agent code generation ability with the one powered by a RAG and discuss the results.   Acknowledgements   A big thank you to SAS Project Anemoi 's members, selflessly sharing their GenAI experiences.   Additional Resources   SWAT Code Generation and Execution in SAS Viya with Azure OpenAI and LangChain: Behind the Scenes. SAS Code Generation and Execution in SAS Viya with Azure OpenAI. GPT-4 Assisted Data Management in SAS Viya: A Custom LangChain Agent Approach. How to Create Your Custom LangChain Agent for SAS Viya. SASPY: Submit workloads to SAS Viya from Python. LangChain Custom Agent. Conversing with Data: Turning Queries into Conversations with SAS Viya, Azure OpenAI and LangChain. Exploring LangChain and Azure OpenAI’s Ability to Write SQL and Join Tables To Answer Questions.   Thank you for your time reading this post. If you liked the post, give it a thumbs up! Please comment and tell us what you think about having conversations with your data. If you wish to get more information, please write me an email.     Find more articles from SAS Global Enablement and Learning here. ... View more

Let's dive into crafting GitLab CI/CD pipelines tailored for SAS Viya. First, you established a GitLab Kubernetes runner. Next, you crafted a bespoke Docker image suited for your SAS Viya ecosystem and uploaded it to a secure, private GitLab container registry. Read this post to understand what a pipeline for SAS Viya looks like, what the different components are and how to put them all together.   Post Series   In this post series you will learn how to:   Register the GitLab Kubernetes Runner. Create Your Own Docker Image for SAS Viya. Use GitLab Pipelines to Execute Commands in SAS Viya, using the Docker image you built as the base for your executor – the current post   Video   Take a look at the following video, in which we demonstrate a GitLab CI/CD pipeline that runs a SAS program in SAS Viya.     GitLab CI/CD Pipeline   The pipeline is designed to submit a SAS program, which is stored in your Git repository, to SAS Viya for execution. This program's task is to load a table into memory within a caslib.   Begin by creating a .gitlab-ci.yml file in your repository's root directory:   default: image: $CI_REGISTRY/$GITLAB_USER_LOGIN/sasgitlabdevops/jass/gitlab-runner-sas:0.07 run-sas-program: stage: build script: - echo "Hello, $GITLAB_USER_LOGIN!" - echo "CI Registry $CI_REGISTRY" # test container folders - ls -la - ls -la /usr/local/bin # SAS Viya CLI - sas-viya plugins list # TLS Certificate location - export SSL_CERT_FILE=/usr/local/bin/gelenv_trustedcerts.pem - echo ${SSL_CERT_FILE} # SAS Viya CLI profile - sas-viya --profile dev profile set-endpoint $dev_endpoint - sas-viya --profile dev profile toggle-color on - sas-viya --profile dev profile set-output fulljson - sas-viya --profile dev profile show - sas-viya --profile dev auth login -user $dev_user -password $dev_passwd # SAS Viya CLI - echo Run a SAS program stored in Git - sas-viya --profile dev batch jobs submit-pgm --pgm-path 1.sas --context default --watch-output --wait-log-list --results-dir ~/   Explanation   Image   default: image: $CI_REGISTRY/$GITLAB_USER_LOGIN/sasgitlabdevops/jass/gitlab-runner-sas:0.07   The configuration snippet you've provided is designed to pull a specific Docker image from your GitLab Container Registry for use in your CI/CD pipeline. This Docker image is the one you previously constructed for your SAS Viya environment.   Here's a breakdown of the image directive in your .gitlab-ci.yml file:   `default:` sets the default configuration for all jobs in the pipeline. `image:` specifies the Docker image to use. $CI_REGISTRY and $GITLAB_USER_LOGIN are placeholders for GitLab's predefined variables; $CI_REGISTRY will become the URL of the GitLab registry, typically registry.gitlab.com. $GITLAB_USER_LOGIN will be replaced with your GitLab username.   The complete image name, including the tag, will be resolved using these variables, ensuring that the correct image is used for the pipeline.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   To learn more about building the Docker image, refer to the associated post.   Executor Pod   The pipeline runs. The Kubernetes runner initiates the execution. The runner creates a temporary Kubernetes executor pod from the pulled Docker image.       Stage, Job and Script   The basic bricks in a GitLab CI/CD pipeline are the stage, which contains jobs. The job contains scripts.   run-sas-program: stage: build script: - echo "Hello, $GITLAB_USER_LOGIN!"   Where:   run-sas-program job runs inside the build stage: script: runs a series of shell scripts: printing variables, listing folders, etc.   SAS Viya CLI commands   List the plugins. Initialize a variable needed by the SAS Viya CLI: the location of the SAS Viya TLS certificates. Recall from step two, we ‘baked in’ the certificate in the Docker image. Create a SAS Viya profile using variables declared when we setup the GitLab runner. Authenticate against SAS Viya (get an access token). Submit a SAS program in batch.   # SAS Viya CLI - sas-viya plugins list # TLS Certificate location - export SSL_CERT_FILE=/usr/local/bin/gelenv_trustedcerts.pem - echo ${SSL_CERT_FILE} # SAS Viya CLI profile - sas-viya --profile dev profile set-endpoint $dev_endpoint - sas-viya --profile dev profile toggle-color on - sas-viya --profile dev profile set-output fulljson - sas-viya --profile dev profile show - sas-viya --profile dev auth login -user $dev_user -password $dev_passwd - sas-viya --profile dev batch jobs submit-pgm --pgm-path 1.sas --context default --watch-output --wait-log-list --results-dir ~/   Notice the variables:   $dev_endpoint is the URL of our SAS Viya environment. $dev_user is the SAS user executing the pipeline. $dev_passwd is the credential for SAS Viya CLI authentication.   These are Project variables, defined under Settings > CI/CD > Variables in GitLab.     From a security perspective you should mask sensitive variables, so that they can’t be printed by accident in a log.     Finally, by using the SAS Viya CLI, you can submit the SAS program to be executed in SAS Viya.    - sas-viya --profile dev batch jobs submit-pgm --pgm-path 1.sas --context default --watch-output --wait-log-list --results-dir ~/     Ultimately, you should see the table successfully loaded in SAS Viya.   Conclusion   In conclusion, this post has walked you through setting up a GitLab CI/CD pipeline designed for SAS Viya, utilizing the Kubernetes runner established in the first step and a custom Docker image crafted in the second step. Looking ahead, we will delve into additional SAS Viya pipelines and various runner options. Thank you for following along!   Acknowledgements   Many thanks to Jan Kostrubiec @JanKostrubiec , Katarzyna Zuk @KatarzynaŻuk , Lars Arne Skår @larsarne , Neil Griffin @Neil-Griffin , from whom I learned the most about DevOps with SAS Viya and GitLab.   Thank you for your time reading this post. If you liked the post, give it a thumbs up! Please comment and tell us what you think about having conversations with your data. If you wish to get more information, please write me an email.     Find more articles from SAS Global Enablement and Learning here. ... View more

  Learn how to create your own self-hosted Azure Pipelines agent. The agent is a Linux Virtual Machine (VM), hosted in Microsoft Azure. This agent performs the build tasks defined in the Azure Pipelines YAML. The YAML can instruct the agent to run a SAS program or SAS Studio flow in batch, import a SAS package, publish a SAS model or a decision to a destination such as Azure or Git.   Overall Picture   The self-hosted agent is the key piece for SAS Viya to work with Azure DevOps. The steps are: Create a virtual machine that acts as a self-hosted agent (this post). Read How to Make Your Virtual Machine Talk with SAS Viya on Azure to learn how to configure communication with the SAS Viya cluster: define Network Security Groups rules, copy certificates, etc. Create an Azure DevOps self-hosted agent. Install software and packages on the self-hosted agent: SAS Viya Command Line Interface CLI, utilities such as pyViyaTools, Python, etc.   Azure Virtual Network Topology   The self-hosted agent is the virtual machine highlighted in the following Azure virtual network VNET topology. For security and many practical reasons, you will have to create not only the virtual machine, but also many other Azure resources.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   You can write Azure CLI directly in the Cloud Shell or from a machine with cloud access. The commands will create Azure resources.   Create Azure Resources   To realize the above topology, I recommend you follow these steps. As you can see, creating the agent Virtual Machine is actually the last step in this process. SSH Keys If You Have the Keys If You Do Not Have the Keys Initialize Variables Create a Resource Group Create a Virtual Network and Subnet Create a Public IP Address Create a Network Security Group Create Network Security Group Rules Create a Virtual Network Interface Card Create an Availability Set Create a Virtual Machine Test SSH to Your VM SSH Keys   To manage the virtual machine you will create, you will need to connect to it using Secure Shell (SSH). To connect using SSH, you will need a set of SSH keys.   If You Have the Keys   First, check you have SSH keys that you can use:   ls -la ~/.ssh # create the private_key variable to match the key name export private_key=~/.ssh/gelazuredm-aks-key   In my case, the key pair was previously generated. It contains: The private key gelazuredm-aks-key The public key gelazuredm-aks-key.pub If You Do Not Have the Keys   If you do not have the keys, you can generate them using the following statement:   cd ~/.ssh ssh-keygen -m PEM -t rsa -b 4096   Name the new keys. You could use a passphrase, which is more secure.   # adapt the private_key variable to match the key name # export private_key=~/.ssh/gelazuredm-aks-key2     Initialize Variables   For the resources you are about to create, it is best to adopt a naming convention. In this example, the resources will be named using the particle $MYUSER:   Create a Resource Group   An Azure resource group is a logical container into which Azure resources are deployed and managed. A resource group must be created before a virtual machine and supporting virtual network resources:   # adapt the private_key variable to match the key name az group create --name $RG --location $LOCATION   Create a Virtual Network and Subnet   Create a virtual network in Azure and a subnet into which you can create your VM. Use az network vnet to create a virtual network named $VNET with the Classless Inter-Domain Routing (CIDR) 10.2.0.0/28 (equivalent to 16 IPs). You also add a subnet named $SUBNET with the CIDR 10.2.0.0/29 (8 IPS). This is the minimum and we really do not need more for our purpose:   az network vnet create \ --resource-group $RG \ --name $VNET \ --address-prefix 10.2.0.0/28 \ --subnet-name $SUBNET \ --subnet-prefix 10.2.0.0/29   Create a Public IP Address   Now let's create a public IP address with az network public-ip create . This public IP address enables you to connect to your VMs from the Internet. Because the default address is static, create with the --allocation-method Static parameter.   az network public-ip create \ --resource-group $RG \ --name $IP \ --allocation-method Static \ --sku Basic   Create a Network Security Group   To control the flow of traffic in and out of your VMs, you apply a network security group to a virtual Network Interface Card (NIC) or subnet. The following example uses az network nsg create to create a network security group named $NSG:   az network nsg create \ --resource-group $RG \ --name $NSG Create Network Security Group Rules   You define rules that allow or deny specific traffic. To allow inbound connections on port 22 (to enable SSH access), create an inbound rule with az network nsg rule create . The following example creates a rule named AllowSSH:   az network nsg rule create \ --resource-group $RG \ --nsg-name $NSG \ --name ${MYUSER}AllowSSH \ --protocol tcp \ --priority 1000 \ --source-address-prefixes 148.172.0.0/16 \ --destination-port-range 22 \ --access allow Note: 148.172.0.0/16 is an IP where employees can connect to the internal network using VPN. Therefore, the rule will allow only organization employees who are connected with corporate VPN to SSH the VM.     Create a Virtual Network Interface Card   Virtual network interface cards (NICs) are programmatically available because you can apply rules to their use. Depending on the VM size, you can attach multiple virtual NICs to a VM. In the following az network nic create command, you create a NIC named $NIC and associate it with your network security group. The public IP address $IP is also associated with the virtual NIC.   az network nic create \ --resource-group $RG \ --name $NIC \ --vnet-name $VNET \ --subnet $SUBNET \ --public-ip-address $IP \ --network-security-group $NSG     Create an Availability Set   Availability sets help spread your VMs across fault domains and update domains. Even though you only create one VM right now, it's best practice to use availability sets to make it easier to expand in the future. Create an availability set for your VM with az vm availability-set create . The following example creates an availability set named $AS:   az vm availability-set create \ --resource-group $RG \ --name $AS   Create a Virtual Machine   You've created the network resources to support Internet-accessible VMs. Now create a VM and secure it with an existing SSH key. In this example, let's create an Ubuntu VM based on the most recent LTS. You can find additional images with az vm image list .   Specify an SSH key to use for authentication. If you do not have an SSH public key pair, you can create them or use the --generate-ssh-keys parameter to create them for you. If you already have a key pair, this parameter uses existing keys in your ~/.ssh folder. Create the VM by bringing all the resources and information together with the az vm create command .   The following example creates a VM named $VM:   az vm create \ --resource-group $RG \ --name $VM \ --location $LOCATION \ --availability-set $AS \ --nics $NIC \ --image UbuntuLTS \ --admin-username $VM_USER \ --ssh-key-value ${private_key}.pub \ --size Standard_B2s     SSH to Your VM   The infrastructure was created. Test the connection, SSH to your VM with the public IP address created:   # SSH connect to your VM JUMPBOXIP=$(az vm list-ip-addresses -g $RG -n $VM --query "[].virtualMachine.network.publicIpAddresses[0].ipAddress" -o tsv) echo "${MYUSER} Jump Box Vm's Public IP: $JUMPBOXIP" cd ~/.ssh ssh -i $private_key jumpuser@$JUMPBOXIP   Conclusion   You created a Virtual Machine that will be configured as a self-hosted agent. Read the next posts to learn how to: Configure communication with the SAS Viya cluster (Network Security Groups rules, certificates). Install software on the self-hosted agent (SAS Viya Command Line Interface CLI, utilities such as pyViyaTools, Python, etc.).   Discussion   There are alternatives to the proposed Azure Virtual Network Topology. The SAS Viya deployment on Azure comes with a pre-deployed virtual machine. You can use it to configure it as a self-hosted Azure Pipelines agent too, without creating a new one.     Read the Next Post in the Series Want to know how to configure this VM to communicate with SAS Viya? Read How to Make Your Virtual Machine Talk with SAS Viya on Azure.   Thank you for your time reading this post. If you liked the post, give it a thumbs up! Please comment and tell us what you think about post content. If you wish to get more information, please write me an email.   Find more articles from SAS Global Enablement and Learning here. ... View more

The second step in this journey involves creating a Docker image that's customized for your SAS Viya environment. After crafting it, the next move is to push this tailor-made Docker image to a GitLab container registry. But what exactly does the Docker image do? How does it integrate with a Kubernetes executor? And what are the benefits of hosting this image in a private GitLab container registry? Explore the post to uncover these answers.   Post Series   In this post series you will learn how to:   Register the GitLab Kubernetes Runner. Create Your Own Docker Image for SAS Viya – the current post. Use GitLab Pipelines to Execute Commands in SAS Viya, using the Docker image you built as the base for your executor.   Summary   For secure execution of SAS Viya jobs using a Kubernetes executor, a Docker image containing essential SAS Viya elements is necessary. This image, which includes the SAS Viya Command Line Interface (CLI) and environment-specific TLS certificates, is pulled by Kubernetes to create an executor pod. The CLI enables task execution and log retrieval within the SAS Viya environment, making the custom Docker image a critical component for operations.   Video   Take a look at the following video, in which we demonstrate the steps to build the Docker image.     Steps   Access Token To authenticate with your private GitLab container registry, you must obtain an access token from your GitLab project. Navigate to 'Settings' and then 'Access Token' to create one. Ensure the token has 'write_repository' permissions for the necessary access or a role that has these permissions (e.g., Developer).   Container Registry Authentication To push a Docker image to a private registry, authentication is required. On a system with Docker installed, use your GitLab credentials to log in to the GitLab registry. For the password, use the access token you created in GitLab.   Docker Image Here is a streamlined version of the instructions:   SAS Viya TLS certificates Create a folder to house the necessary files. Obtain the certificates using a kubectl command.   mkdir -p docker_gitlab cd docker_gitlab mkdir -p files # Obtain the Truststore Certificate File from Your Cluster kubectl -n name-of-namespace cp $(kubectl -n name-of-namespace get pod | grep "sas-logon-app" | awk -F" " '{print $1}'):security/trustedcerts.pem files/gelenv_trustedcerts.pem # Copy the certificate in /files ls -la ls -la files     The SAS Viya CLI file For Linux, download the TGZ file from the SAS Downloads Page in the docker_gitlab/files folder.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.     Dockerfile Within the docker_gitlab folder, craft a Dockerfile.   tee Dockerfile> /dev/null << EOF FROM centos:8 MAINTAINER YourName “your email” RUN sed -i 's/mirrorlist/#mirrorlist/g' /etc/yum.repos.d/CentOS-* RUN sed -i 's|#baseurl=http://mirror.centos.org|baseurl=http://vault.centos.org|g' /etc/yum.repos.d/CentOS-* RUN dnf update -y && dnf upgrade -y \ && dnf install epel-release -y \ && dnf install jq -y \ && dnf install -y nfs-utils \ && dnf clean all \ && rm -rf /var/cache/yum ADD files/sas-viya-cli-1.21.30-linux-amd64.tgz /usr/local/bin WORKDIR /usr/local/bin COPY ./gelenv_trustedcerts.pem . RUN sas-viya plugins install --repo SAS audit \ && sas-viya plugins install --repo SAS authorization \ && sas-viya plugins install --repo SAS batch \ # && sas-viya plugins install --repo SAS cas \ # && sas-viya plugins install --repo SAS compute \ && sas-viya plugins install --repo SAS configuration \ # && sas-viya plugins install --repo SAS dcmtransfer \ # && sas-viya plugins install --repo SAS decisiongitdeploy \ # && sas-viya plugins install --repo SAS devices \ # && sas-viya plugins install --repo SAS folders \ # && sas-viya plugins install --repo SAS fonts \ # && sas-viya plugins install --repo SAS identities \ && sas-viya plugins install --repo SAS job \ # && sas-viya plugins install --repo SAS launcher \ # && sas-viya plugins install --repo SAS licenses \ # && sas-viya plugins install --repo SAS mip-migration \ # && sas-viya plugins install --repo SAS models \ # && sas-viya plugins install --repo SAS notifications \ # && sas-viya plugins install --repo SAS oauth \ # && sas-viya plugins install --repo SAS qkbs \ # && sas-viya plugins install --repo SAS reports \ # && sas-viya plugins install --repo SAS rtdmobjectmigration \ # && sas-viya plugins install --repo SAS scoreexecution \ # && sas-viya plugins install --repo SAS sid-functions \ && sas-viya plugins install --repo SAS transfer \ # && sas-viya plugins install --repo SAS workload-orchestrator \ && sas-viya plugins list CMD ["/bin/bash"] EOF   Building a Docker image requires specific files, including the SAS Viya TLS certificates and the SAS Viya CLI file, which can be obtained from the support.sas.com website. Let's examine the Dockerfile steps:   The process begins by pulling the base image. While we use a CentOS image, alternative modern base images like RedHat's ubi8, which serves as the foundation for many SAS container images, can also be used. The next step updates and upgrades the existing packages on the system, followed by the installation of additional repositories, tools, and utilities. After setting up the environment, we incorporate the core SAS Viya components by adding the TLS certificates, and then we proceed to install the SAS Viya CLI. We decided to install only a few plugins, to keep the image size light.   Docker Build Proceed to build the Docker image. Assign a tag to the newly created image. Push the tagged image to your private GitLab container registry.   GITLAB_REGISTRY=registry.gitlab.com/${YOUR_GITLAB_USER}/${YOUR_GITLAB_PROJECT} TAG=0.07 # Build from Dockerfile docker build -t ${GITLAB_REGISTRY}/jass/gitlab-runner-sas:${TAG} . # List images docker image ls # Push to GitLab container registry docker push ${GITLAB_REGISTRY}/jass/gitlab-runner-sas:${TAG}   Discussion   Why a custom image? The question of using a custom image versus a pre-existing SAS Viya CLI Docker image is valid. While it's true that there is a SAS Viya CLI Docker image available, it's primarily suited for use with a shell executor, not directly with a Kubernetes executor.   For Kubernetes executors, which operate within a pod, having the TLS certificates accessible is essential, as well as the SAS Viya CLI image. There may be alternative methods to achieve this, but they would require further investigation.   Why a private GitLab container registry? As you have the TLS certificate “baked in” the image, you must keep that image private. You can’t put it on a public registry as you would expose your SAS Viya certificates.   That's exactly the reason for using a private GitLab container registry, a decision rooted in security.   Conclusion   In this post, we've guided you through the steps to create a Docker image customized for your SAS Viya environment and demonstrated how to push the custom Docker image to a GitLab container registry.   Moving forward, we'll explore how to securely run SAS Viya jobs using GitLab CI/CD pipelines.   Thank you for reading!   Acknowledgements Many thanks to Jan Kostrubiec @JanKostrubiec, Katarzyna Zuk @katarzynazuk, Lars Arne Skår @larsarne, and Neil Griffin @Neil-Griffin, from whom I learned the most about DevOps with SAS Viya and GitLab.   Thank you for your time reading this post. If you liked the post, give it a thumbs up! Please comment and tell us what you think about having conversations with your data. If you wish to get more information, please write me an email.     Find more articles from SAS Global Enablement and Learning here. ... View more

The purpose of this post series is to guide you to configure GitLab CI/CD pipelines for effective operation with SAS Viya. The first step on this journey is to register a GitLab runner and define an executor. What is the role of the GitLab runner, what is an executor and why do you need one for triggering SAS Viya workloads? Read the post to find out more.     Post Series   In this post series you will learn how to:   Register the GitLab Kubernetes Runner – the current post. Create Your Own Docker Image for SAS Viya. Use GitLab Pipelines to Execute Commands in SAS Viya, using the Docker image you built as the base for your executor.   Before diving in the details, let’s answer the following question:     What's the Objective Here?   Implementing GitLab CI/CD pipelines brings numerous benefits to SAS Viya operations. These include automating processes, code versioning, fostering collaboration in development, facilitating content transfer across various environments, storing as well as archiving artifacts.   GitLab CI/CD pipelines empower you to:   Automate segments of a business process. Execute SAS and Python programs from a version-controlled GitLab repository. Import SAS packages from a JSON file within the same repository. Initiate a SAS Studio flow or execute a SAS job. Export SAS packages into JSON files. And accomplish much more.     Video   Take a look at the following video, in which a GitLab Kubernetes runner is registered and tested.       GitLab Concepts   After the video, let's familiarize ourselves with some GitLab concepts:   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.     GitLab Job: This is the smallest component of a pipeline, which contains one or more commands that need to be executed, such as to run a SAS program in SAS Viya. The jobs and the commands are defined inside a file .gitlab-ci.yml.   GitLab Runner: This is an agent installed on a server different from the GitLab server. The GitLab runner receives instructions from the GitLab server on which jobs to run. Each runner must be registered with the GitLab server.   Executor: Each runner requires at least one executor. An executor is the environment where the job will be executed.   Thus, the crucial distinction is: The GitLab Kubernetes runner acts as the agent that oversees and delegates the tasks, whereas the Kubernetes executor is the environment where these tasks are actually carried out.   GitLab offers various executors that you can use depending on your needs. These include Shell, Kubernetes, Docker, SSH, VirtualBox, Parallels, and Docker Machine.     Executors   In this post we will focus on a Kubernetes executor. Why Kubernetes?   We utilize a Kubernetes runner and executor since SAS Viya is deployed within a Kubernetes cluster. The Kubernetes executor pod is temporary, existing only for the duration of the job execution. Utilizing a Kubernetes executor is advantageous in this context. The runner spawns an executor pod using a Docker image, which can be sourced from a private container registry linked to your GitLab project or a public one like Docker Hub.     Setting Up a GitLab Runner   There are five simple steps to create the GitLab runner:   Create the runner. Store the runner authentication token. Deploy and start the runner. Verify the runner registration. Run a test job.     1. Create the runner   In your GitLab repository, perform the following steps:   Navigate to Settings -> CI/CD and expand the 'General pipelines' section. Under 'Git strategy', set the 'Git shallow clone' field to 0. This has been found by my SAS colleagues to optimally support pipeline functioning. Expand the 'Runners' section, switch to the 'Instance runners' tab and deactivate instance runners. Generate a new project runner. Select 'Linux', add a tag, enable 'Run untagged jobs', designate the runner name, and set the 'Maximum job timeout' to a value superior to 600. Finally, create the Runner. Enabling 'Run untagged jobs' permits the runner to execute all pipelines, not just those specifying the tag. With the above option disabled, if your aim is to execute certain pipelines on the runner, it's necessary to incorporate the tag within the pipeline code.     2. Store the runner authentication token   Copy the value of the runner authentication token and save it somewhere safe.   GITLABTOKEN=glrt-....     3. Deploy and start the runner   For clear workload separation, the runner will be deployed in a fresh Kubernetes namespace. Executors will operate as pods within this specifically assigned workspace.   kubectl create namespace gitlab   A values.yaml file allows you to customize some of the runner pod settings, such as pods resources, number of replicas. You can find the full list of variables values.yaml.   cd ~ tee values.yaml > /dev/null << EOF ## Configure resource requests and limits for the runner pods resources: limits: memory: 1024Mi cpu: 750m requests: memory: 512Mi cpu: 500m ## How many runner pods to launch replicas: 2 EOF   The official way of deploying a GitLab runner instance into your Kubernetes cluster is by using the gitlab-runner Helm chart. This chart configures a GitLab runner to run using the Kubernetes executor. For each new job it receives from GitLab CI/CD, it provisions a new pod within the specified namespace to run it.   GITLABTOKEN=glrt-.... helm repo add gitlab https://charts.gitlab.io helm install \ -f values.yaml \ --namespace gitlab gitlab-sasviya-008 \ --set gitlabUrl=https://gitlab.com/,runnerRegistrationToken=$GITLABTOKEN \ --set rbac.create=true \ --set runners.privileged=true \ --set checkInterval=5 \ --set nodeSelector."kubernetes\.io/role"=agent \ gitlab/gitlab-runner   Explanation of the options:   'helm repo' - Adds the GitLab Chart to Helm. 'helm install' - Deploys the runner. '-f values.yaml' - Includes the settings defined in your custom 'values.yaml' file. '--namespace gitlab' - Specifies the Kubernetes namespace. 'gitlab-sasviya-008' - Links with the runner name we registered in the GitLab interface. 'gitlabUrl' - Points to the full URL where your GitLab project is hosted. In some organizations, GitLab can be deployed on a company subdomain. 'runnerRegistrationToken' - Indicates the token generated when you registered the runner. The token allows the runner to authenticate in GitLab. '--set rbac.create' - If your cluster has RBAC enabled, you can opt to have the chart create the service account for you. '--set runners.privileged' - Allows the GitLab Runner to run using privileged containers. This might be necessary if you need to use the Docker executable within your GitLab CI/CD jobs. '--set checkInterval' - Defines in seconds how often to check GitLab for new builds. '--set nodeSelector."kubernetes.io/role"=agent' - Labels the GitLab Runner pods.   For more general information about Helm, see Helm basics.     4. Verify the runner registration   To verify the registered runners, navigate to Settings -> CI/CD and click on 'Expand' in the runners section.     5. Run a test job   Create a new file .gitlab-ci.yml in your GitLab repository, and paste the following inside:   show-hello: script: - echo "hello world"   Commit the changes with a comment. Go to Build -> Pipelines and check the status.     Conclusion   In this post, we've walked through the process of setting up a GitLab runner for your GitLab repository in a Kubernetes cluster. This is an essential first step towards seamless CI/CD operations with SAS Viya.   In the following steps, you'll construct a Docker image that your Kubernetes executor will use to send commands to SAS Viya and write a few SAS Viya CI/CD pipelines.   Thank you for your time!     Acknowledgements   Many thanks to @JanKostrubiec, @katarzynazuk, @larsarne, and @Neil-Griffin from whom I learned the most about DevOps with SAS Viya and GitLab.     Thank you for your time reading this post. If you liked the post, give it a thumbs up! Please comment and tell us what you think about having conversations with your data. If you wish to get more information, please write me an email.     Find more articles from SAS Global Enablement and Learning here. ... View more

The business glossary is not just an add-on, but an indispensable tool in any data catalog. Starting from SAS Viya release LTS 2024.03 (May 2024), the SAS Information Catalog has introduced a brand new glossary. It facilitates a common understanding of data elements in an organization by defining terms in a clear, accessible language. The glossary eliminates ambiguity in business terminology, fostering consistent communication throughout the organization. To discover a summary of the key functionalities, please read more.   Why is the Glossary Essential?   Consider non-cataloged data as a chaotic pile of unsorted books. Would you want to be the person desperately searching for a specific piece of information in this random heap of data?   AI-generated image: ChatGPT 4o.   In contrast, cataloging data is akin to neatly organizing these books onto shelves. Wouldn't you rather be the individual who can effortlessly scan through a well-arranged library to find what they need?   AI-generated image: ChatGPT 4o.   This is where the glossary steps in. Acting as a well-structured library, it systematically arranges, groups, and labels cataloged data, making it easy for users to locate the exact information they require.   AI-generated image: ChatGPT 4o.   What Makes Up the Glossary?   The glossary is built on terms and term types.   Terms A term is the fundamental building block in the glossary. It helps you capture the essential information.     What attributes you can capture is defined by a term type.   Term Types Term types function as templates, aiding in the organization of terms into meaningful functional groups based on your business needs. This structure allows for customization and capturing specific details.     Linking Terms to Data Assets Terms can be attached to columns of cataloged data assets.       How to Use the Glossary? The glossary is flexible and adaptable, with terms structured in hierarchies, providing an immediate overview of the information.     The hierarchy can be organized based on business units, processes, information catalogs, or data views. This versatile structure aligns with your organization's specific data governance objectives.   Terms can be organized using various structures, such as:   Organizational View: This follows the structure of Business Unit > Department > Business Term. Process View: This is arranged as Business Process > Process Step > Business Term. Information Catalog View: This is set up as Subject Area > Domain > Business Term. Data View: This adopts the structure of Area > Entity > Attribute.   Who Can Use the Glossary?   The glossary is accessible to catalog users and glossary administrators.   The catalog users can consult the terms, attach data asset columns to a term, while glossary administrator can manage term types, create, update and delete terms and even import terms from a CSV file.   The Glossary REST API allows you to manage all aspects of the glossary from the comfort of a command line or a program written in SAS, Python. The Glossary REST API is a great tool that can help you automate all things related to a glossary.   The Glossary REST API provides the convenience of managing all aspects of the glossary directly from a command line, a shell script, or a program written in SAS or Python. This powerful tool facilitates automation of all glossary-related tasks.   The Bigger Picture   The glossary forms an integral part of the catalog. It works in conjunction with other features like asset tags, descriptions, and contacts to provide a comprehensive and clearer picture of the data. It is not a standalone tool but a key part of the data governance ecosystem.     Conclusion   In conclusion, the glossary in SAS Information Catalog is a vital tool for data governance and data management. It adds a semantic layer of knowledge in natural language, making data interpretation faster and more accurate. Its integration with other catalog functionalities like asset tags, descriptions, and contacts further enhances data understanding.   Thank you for your time reading this post. If you liked the post, give it a thumbs up! Please comment and tell us what you think about having conversations with your data. If you wish to get more information, please write me an email.   More SAS Information Catalog Posts Boost Data Governance with SAS Information Catalog OpenMetadata Connector. Discover Your Data with SAS Information Catalog APIs from Python – Access. Leveraging SAS® Information Catalog REST APIs: Programmatically Discovering Data. Download Metadata and Metrics with SAS Information Catalog APIs from Python. Discover Your Data with SAS Information Catalog APIs from Python – Upload to CAS. Using Azure OpenAI's GPT-4 to Query Cataloged Assets (combining LLMs and Catalog Instances APIs).   Find more articles from SAS Global Enablement and Learning here. ... View more

As of the 2024.01 version, the SAS OpenMetadata connector can be utilized for the automatic addition of SAS Information Catalog metadata into an OpenMetadata repository. In this article, we'll present a brief video guide on how the SAS Information Catalog assets appear in OpenMetadata. We’ll take a look at the steps to achieve this integration.   What Assets?   The OpenMetadata repository can include metadata from the following catalog assets:   SAS libraries SAS tables. SAS columns. CAS libraries. CAS tables. SAS Studio flows. SAS Visual Analytics reports.   However, this connector does not support writing metadata back to the SAS Information Catalog repository.       Why OpenMetadata Integration?   SAS customers have long sought a method to integrate SAS assets with those from other applications. OpenMetadata provides various connectors for this purpose. Now, it's possible to export assets from SAS to OpenMetadata. Here, assets from different applications can be linked using tags, terms, domains, or manual lineage, among other options.       Steps   Procedure to Incorporate SAS Information Catalog Assets into OpenMetadata:   Initiate OpenMetadata Deployment. Options: Docker deployment (I had more success with it). Kubernetes deployment (I had some issues with port forwarding). Set up the SAS Open Metadata Connector. Import Metadata from SAS Information Catalog. ExploreMetadata.   This article will primarily concentrate on metadata exploration. Subsequent posts will delve into details about deployment, connection, and the ingestion process.       Watch the Short Video     Conclusions   The OpenMetadata connector for the SAS Information Catalog facilitates enhanced asset integration, including assets from various catalogs. In the face of numerous compliance and data protection regulations, data governance features like lineage, tagging, attaching terms, policies, and owners to assets are beneficial.   They help solve a few problems. These features in OpenMetadata allow for the creation of a comprehensive map detailing the assets and their interactions in data management processes.   Discussion   Personally, I'm delighted to see that SAS has opted for an open-source product for external asset integration. This product is easy to set up and features numerous connectors. Furthermore, Airflow efficiently manages the metadata ingestion process.   Please comment and let us know what you think about OpenMetadata, the integration, what problems it solves, what problems it doesn't solve, what the future direction should be. It's always nice to hear your opinion and personal experience.   What to Read Next   SAS Information Catalog: Administrator’s Guide OpenMetadata SAS Connector. More SAS Information Catalog Posts Boost Data Governance with SAS Information Catalog OpenMetadata Connector. Discover Your Data with SAS Information Catalog APIs from Python – Access. Leveraging SAS® Information Catalog REST APIs: Programmatically Discovering Data. Download Metadata and Metrics with SAS Information Catalog APIs from Python. Discover Your Data with SAS Information Catalog APIs from Python – Upload to CAS. Using Azure OpenAI's GPT-4 to Query Cataloged Assets (combining LLMs and Catalog Instances APIs).   Thank you for your time reading this post. If you liked the post, give it a thumbs up! Please comment and tell us what you think about having conversations with your data. If you wish to get more information, please write me an email.     Find more articles from SAS Global Enablement and Learning here. ... View more