SAS Viya Monitoring for Kubernetes has introduced two features in version 1.2.36 (15th April 2025) aimed at simplifying deployment; auto-generating Ingress definitions and automatically specifying storage class references. These enhancements reduce the need for manual configuration, making setup easier and more efficient.   How Ingress is Used in SAS Viya Monitoring   Ingress resources in Kubernetes manage external access to services within the cluster, typically HTTP and HTTPS traffic. In SAS Viya Monitoring, Ingress facilitates access to web-based monitoring tools like Grafana and OpenSearch Dashboards (OSD). Properly configured Ingress ensures that users can securely and efficiently interact with these interfaces.   There are two primary approaches to Ingress configuration:   Host-Based Ingress: Each application is accessed via a unique subdomain. Example: - Grafana: https://grafana.gelcorp.com/   - OpenSearch Dashboards (OSD): https://osd.gelcorp.com/     Path-Based Ingress: A single domain is used, with applications differentiated by URL paths. Example: - Grafana: https://monitoring.gelcorp.com/grafana   - OpenSearch Dashboards (OSD): https://monitoring.gelcorp.com/osd     The choice between host-based and path-based Ingress depends on your organisation's policies. Detailed guidance on configuring Ingress for SAS Viya Monitoring is available in the official documentation.   Auto-Generated Ingress Definitions   Previously, configuring Ingress resources required manual editing of YAML files. With this new feature, users can automate the creation of these definitions by setting environment variables, preferably by including them in the $USER_DIR/user.env file. See the Pre-Deployment topic in the Help Center documentation for information on the $USER_DIR directory and more information about the SAS Viya Monitoring customisation process. The system supports both host-based and path-based Ingress configurations, aligning with common deployment scenarios.   To enable auto-generated Ingress definitions:   Set the environment variable AUTOGENERATE_INGRESS=true. Specify your Ingress type by setting ROUTING to either host or path. Specify the hostname to be used in the URLs by setting the environment variable BASE_DOMAIN. This is required. Apply your configuration as usual, and the system will generate the appropriate Ingress definitions automatically.   If you provide your own TLS certificate files, it's also possible for the autogeneration process automatically create corresponding Kubernetes secrets needed by the Ingress resources. Two further variables are required for this:   Set INGRESS_CERT to point to the location of the TLS certificate file. Set INGRESS_KEY to the TLS key file.   How Storage Classes Are Used in SAS Viya Monitoring   Persistent storage is crucial for retaining monitoring data, logs, and metrics across pod restarts and deployments. Kubernetes uses Persistent Volume Claims (PVCs) to request storage resources, and storage classes define the types of storage available for these PVCs.   In SAS Viya Monitoring, various components require persistent storage to function correctly. For instance, Prometheus needs storage to retain metric data, and OpenSearch requires storage for log data. Configuring appropriate storage classes ensures that these components have access to reliable storage resources.   Key considerations for storage configuration include:   Durability: Ensure that data persists beyond the lifecycle of individual pods by using persistent volumes external to the Kubernetes cluster. This approach safeguards data integrity during scaling or pod failures. Performance: Select storage classes that meet the performance requirements of your monitoring workloads, considering factors like input/output operations per second (IOPS) and latency. Capacity Planning: Allocate sufficient storage capacity to accommodate the volume of monitoring data generated, with appropriate provisions for future growth.   For comprehensive information on storage requirements and configuration in SAS Viya Monitoring, refer to the official documentation.   Automatic Storage Class References   Specifying storage classes for PVCs traditionally involved modifying YAML files manually. The latest update introduces an automated approach, generating the necessary storage class references based on predefined environment variables, preferably by including them in the $USER_DIR/user.env file. See the Customization topic in the Help Center documentation for information on the customisation process. While this simplifies deployment for standard configurations, manual customisation remains an option for more complex setups.   To enable automatic storage class references:   Set the environment variable AUTOGENERATE_STORAGECLASS=true. Define your preferred storage class by setting STORAGECLASS to the appropriate value for your environment. It is important to note that the storageClass specified must exist; SAS Viya Monitoring will not create any storageClass resources. The system will apply this storage class to all relevant PVCs at deployment time without requiring manual edits.   Next Steps   These new capabilities provide a streamlined way to configure Ingress and storage in SAS Viya Monitoring. By reducing manual effort and automating common setup tasks, they make deployment more efficient while still allowing for customisation where needed. Refer to the the Help Center for more information about customising the process to suit your needs.   To get started, ensure you have yq version 4.32.2 or later (which will be a requirement for future releases of SAS Viya Monitoring for Kubernetes). Detailed instructions, including configuration examples, can be found in the documentation.   My thanks to Greg Smith for his contributions to this post.     Find more articles from SAS Global Enablement and Learning here. ... View more

The SAS Viya Orders CLI, the command-line tool for managing SAS Viya software order assets, has recently introduced a new feature that enhances control and precision when downloading deployment assets. As of the latest version (1.7.0, released February 2025) available from the GitHub repository, users can now specify a cadence release value when downloading deployment assets. This addition addresses a use case you may have encountered when deploying or updating your software. Let’s check out how it works and why it matters.   The Problem: Hidden Patches in the Same Version Number   Imagine this scenario: On March 27th, a new SAS Viya version is released. You download the deployment assets and deploy them to a test environment. After running your tests, everything looks good, and you greenlight the deployment to production on March 30th.   Between March 27th and March 30th, SAS releases several patches for that version. The version number remains unchanged, and on the surface, there’s no obvious indication that the assets have been updated to include those new patches. You download the artifacts again on March 30th, expecting the same package you tested, only to find that the patched version is different. This discrepancy might go unnoticed until the production system behaves in an unexpected manner or generates some form of error.   The Solution: Specifying cadence release in the CLI   The new feature in the SAS Viya Orders CLI allows users to explicitly specify a cadence release, which denotes the 'patch version', when downloading deployment assets. This means you’re no longer constrained to downloading the latest patched version of SAS Viya, which was previously the default behaviour. Instead, you can pinpoint the exact release you want, matching what you tested, and avoid surprises in production.   This functionality mirrors capabilities in the SAS Viya Deployment Operator and sas-orchestration tool, where users can define a specific cadence release to ensure consistency across environments. By bringing this precision to the SAS Viya Orders CLI, SAS empowers users to maintain control over their deployment process, aligning with real-world needs where testing and production must stay in sync.   How It Works   The updated CLI now includes an option to specify the cadence release alongside the cadence name and version when using the deploymentAssets command. The exact syntax might look something like this (check the latest README.md for confirmation):   viya4-orders-cli deploymentAssets / <order-number> / <cadence-name> / <version-number> / <cadence-release-number>   Here’s a breakdown:   <order-number> : Your specific SAS Viya order number. <cadence-name>: The release cadence (e.g., Stable, LTS). <cadence-version>: The version number (e.g., 2025.03). <cadence-release-number>: The new value that identifies the exact patch level or release ID you want.   For example, if you tested 2025.03 with a cadence release of 20250327*, you could ensure you download that exact artifact for production, even if patches have since incremented the release to, say, 20250330.   * Simplified example only; actual release numbers have an epoch date-time value like 20250326.1721414421837.   Why This Matters   This new feature tackles several key challenges:   You can now guarantee that what you tested is what you deploy, avoiding the risk of untested patches appearing in production environments. No more guessing whether a release has been patched. You decide which artifact to pull based on your needs. For organisations with strict compliance requirements, this ensures you can track and replicate/automate the exact software state across deployments. You can still get the latest patch version by leaving cadence release unspecified.   This level of control eliminates the disconnect between testing and deployment, making the process smoother and more predictable.   Final Thoughts   The addition of the cadence release option in the SAS Viya Orders CLI is a useful feature for anyone managing SAS Viya deployments. It addresses a real-world need, balancing the need to stay up-to-date with software patches with the stability required for production environments.     Find more articles from SAS Global Enablement and Learning here. ... View more

SAS Viya continues to evolve, offering organisations flexibility in how they manage software updates. As part of this ongoing refinement, changes have been introduced to update paths for the Long-Term Support (LTS) cadence. These adjustments impact how customers can perform updates and maintain their SAS Viya LTS environments. By staying informed and aligning update strategies with business needs, customers can maximise the value of SAS Viya while minimising potential issues and disruptions.   Reviewing The Two Update Cadences   Long-Term Support (LTS)   Designed for stability and predictability. Updates are less frequent but include critical fixes and security patches. Ideal for enterprises that require a consistent and well-tested environment over an extended period.   Stable   Provides access to the latest features and improvements more frequently. Allows customers to take advantage of innovations as they become available. Suitable for organisations that prioritise agility and staying up to date with the latest advancements in SAS Viya.   What’s Changing?   Recent adjustments to the update process affect how customers can move to newer LTS versions. These changes aim to reduce the likelihood of problems and ensure smoother transitions when updating software.   Update paths are documented for supported versions: The SAS Viya Platform Administration Guide's Versions in Standard Support page explicitly lists support levels for versions of SAS Viya as well as supported update pathways. Check the documentation prior to each update, to ensure you are aware of latest versions released as well as those dropped from standard support.     Example upgrade paths for 2025.02    Select this image to see a larger version. Mobile users: To view the image, select the "Full" version at the bottom of the page.   LTS Update Paths: A new policy has been officially adopted, requiring customers to follow a sequential update process when updating LTS versions. Skipping LTS releases is no longer supported. For example, customers on LTS 2023.03 must update to LTS 2023.10, then LTS 2024.03, and then LTS 2024.09 in sequence. Each update must follow pre-update checklists, review deployment notes, and assess the What’s New documentation to ensure compatibility and a smooth transition.   Kubernetes Version Considerations: Customers must verify Kubernetes version compatibility when updating SAS Viya. Certain Kubernetes version required by LTS releases may be unsupported or on extended support for a given cloud provider.   What This Means for Customers   Organisations should review their update strategies in light of these changes. Whether your priority is long-term stability or continuous feature updates, understanding the refined update paths will help ensure a seamless experience with upgrading your SAS Viya deployment.   For full details on these changes, including technical requirements and best practices, refer to the updated documentation.   My thanks to Edie Jeffreys for her contributions.       Find more articles from SAS Global Enablement and Learning here. ... View more

There are a number of useful features, settings and tools that administrators can utilise to audit user activity in SAS Viya 3.x. In this post, we will explore the options available for the auditing of report actions in SAS Visual Analytics.   Important   Before reading on, it should be stressed that there may be a negative impact on system performance as a result of modifying the auditing configuration to capture more detailed records. Broadening the scope of the auditing configuration will generate considerable amounts of data, which will result in a level of slowness, as well as increased disk consumption. If you encounter poor performance after changing the configuration, you may need to reduce the scope by limiting the applications, actions or resources for which to capture audit records.   Overview   Auditing in SAS Viya is based on the SAS Operations Infrastructure, which is based on SAS's event-driven architecture. The event framework uses RabbitMQ as the message broker for processing all events generated by SAS Viya services.   The default behaviour is to capture four basic actions (create, read [failure], update and delete) on report objects. This can be modified using the configuration properties for the Audit service's sas.audit.record configuration instance in SAS Environment Manager.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   When processing an event, the Audit service uses the properties defined here to determine whether it is captured in the audit records.     Configuration properties must be defined in a specific pattern. The following are valid options for configuring the audit service to capture records on a per application basis:   application-name.enabled application-name.event-action-type.enabled application-name.event-action-type.state   where application-name refers to the service name (for example, reports, identities or compute) and event-action-type is one of the action properties from an internal event action registry. State refers to the success or failure of the event action.   Additionally, the configuration properties can be set to capture actions based on entry type rather than by application. For example, security actions can be captured by setting security.action.login.state=all in the type property.   Audit records are written to the SAS Infrastructure Data Server. The data is not loaded to the CAS table that drives the pre-canned audit reports in Environment Manager until the Operations Infrastructure's genAudit task is run (every 2 hours by default). We can use the sas-admin CLI's audit plugin to view the audit records in PostgreSQL immediately. Keep reading for some examples.   Records older than 7 days in the datamart are archived to a specified location on disk. The sas.audit.archive configuration instance determines the rules for the archiving of audit records.   Auditing Report Access   How can we tell what reports our user (Ahmed) opened? Well, remember that audit records are only created for failed read attempts by default due to the setting of resource.action.read.state=failure in sas.audit.record. We can update this value to all to capture successful as well as failed attempts for all resources, but note that this is generally not recommended due to the impact on performance as a result of the amount of data generated. I set this option in my lab for the purpose of this demo, as I am the only user and it is a throwaway environment. After doing so, I ran the CLI command again, this time filtering for entries created by the reports application only. /opt/sas/viya/home/bin/sas-admin --output text audit list --user-id Ahmed --application reports   ID Time Stamp Action State User ID Application URI 94a2d4ba-aefc-4d71-9961-04ada7274e91 2019-10-02T02:31:05.253Z read success Ahmed reports /reports/reports/cbf97b0a-457d-4b4f-8913-547e0cdf390c b40b1ecc-8cd9-4733-99a7-c3e63e06adaf 2019-10-02T02:30:53.703Z read success Ahmed reports /reports/reports/cbf97b0a-457d-4b4f-8913-547e0cdf390c e57a7972-4b5c-4df9-90a6-ed038a4e74cb 2019-10-02T02:30:52.806Z read success Ahmed reports /reports/reports/cbf97b0a-457d-4b4f-8913-547e0cdf390c c71d2094-d18e-4cee-8e03-6e8c7505d070 2019-10-02T02:30:10.382Z read success Ahmed reports /reports/reports/cbf97b0a-457d-4b4f-8913-547e0cdf390c cb10dfb4-bf7d-40f8-a709-456abff091d9 2019-10-02T02:30:00.277Z read success Ahmed reports /reports/reports/cbf97b0a-457d-4b4f-8913-547e0cdf390c 59735b62-d8f4-4f91-9a15-b447f76bb483 2019-10-02T02:16:54.625Z read success Ahmed reports /reports/reports/cbf97b0a-457d-4b4f-8913-547e0cdf390c   Interestingly, there are multiple read entries for the same report. The reason is that these reads represent more than just the action of opening of a report. In fact, some of these records are created when a report is selected in the Content area of SAS Environment Manager, or when thumbnails for the reports are rendered in the list of Recent reports in Report Viewer and SAS Drive (documented in SAS Note 62355). To identify the record that represents the actual opening of the report, we need another option in our command. /opt/sas/viya/home/bin/sas-admin --output text audit list --user-id Ahmed --application reports --details   ID Time Stamp Type Action State Description User ID Application Remote Address URI 94a2d4ba-aefc-4d71-9961-04ada7274e91 2019-10-02T02:31:05.253Z resource read success Ahmed reports 192.168.1.2 /reports/reports/cbf97b0a-457d-4b4f-8913-547 b40b1ecc-8cd9-4733-99a7-c3e63e06adaf 2019-10-02T02:30:53.703Z resource read success Ahmed reports 192.168.1.1 /reports/reports/cbf97b0a-457d-4b4f-8913-547 e57a7972-4b5c-4df9-90a6-ed038a4e74cb 2019-10-02T02:30:52.806Z resource read success Ahmed reports 10.96.17.168 /reports/reports/cbf97b0a-457d-4b4f-8913-547   Adding the details option displays IP addresses in the output. The one that represents the report open is the one that containing the IP address of the client machine. In this example, that would be 10.96.17.168 (the record on the bottom).   We can use the CLI to display more detailed information about an audit record by running the audit plugin's show-info command. /opt/sas/viya/home/bin/sas-admin --output text audit show-info --id 94a2d4ba-aefc-4d71-9961-04ada7274e91   ID 94a2d4ba-aefc-4d71-9961-04ada7274e91 Time Stamp 2019-10-02T02:31:05.253Z Type resource Action read State success User ID Ahmed Remote Address 192.168.1.2 Trace ID 24b3c0cac3add8e3 Application reports URI /reports/reports/cbf97b0a-457d-4b4f-8913-547e0cdf390c Folder /Products/SAS Visual Analytics/Samples Report Name Retail Insights HTTP Status Code 200 HTTP Method GET   As indicated in the SAS Note, only reports under the /Products and /Public folders produce audit records by default. Follow the steps in the note to modify the configuration to capture audit records for reports in all locations.   Auditing Report Actions   For auditing other report actions, such as filtering, exporting, printing, sharing reports, it's a little more complicated.   Filtering   Applying filters to your report will produce more audit records similar to the below from the reportData application, but there is currently no simple way to determine exactly how the report was filtered.   81d4100b-92e1-4dc2-9464-1fe102a062a9 2019-10-02T05:07:24.771Z create success Ahmed reportData /reportData/jobs/0c3691a3-7c55-4774-af19-684a1eda8641_c36 de376ccb-30ee-4ec5-90d7-dfad2b548d19 2019-10-02T05:07:25.901Z create success Ahmed reportData /reportData/jobs/0c3691a3-7c55-4774-af19-684a1eda8641_c35 8a632ec8-e145-4c1e-ad46-69e377270ecf 2019-10-02T05:07:25.943Z read success Ahmed reportData /reportData/results/0c3691a3-7c55-4774-af19-684a1eda8641_0c3691a3-7c55-4774-af19-684a1eda8641_c35/files/dd27639.csv b33a90b2-5ba1-4cda-aeb7-aeb42356f358 2019-10-02T05:07:25.949Z read success Ahmed reportData /reportData/results/0c3691a3-7c55-4774-af19-684a1eda8641_0c3691a3-7c55-4774-af19-684a1eda8641_c35/files/dd27639index.csv   Exporting   The transfer service is responsible for handling the export of content. Therefore, assuming resource.action.read.enabled is set to its default value of failure, we need to turn on auditing for the transfer service. We can do so by adding these new entries to the application property in the sas.audit.record configuration instance.     This will capture all CRUD events (including successful reads) for the transfer service only. Now, when exporting a report from SAS Environment Manager's Content area, audit records similar to the below are captured.   51f68e79-000c-4f82-8015-dcaebdc6552e 2019-09-25T13:25:52.719Z resource create success sasadm transfer 10.96.10.154 /transfer/exportJobs/7f1bbfc7-fdfd-4f09-8527-97fb071746e0 ... 82c18db3-414d-4cde-8b0a-6e76cb3fc315 2019-09-25T13:25:54.823Z resource read success sasadm transfer 10.96.10.154 /transfer/packages/85d85fac-f7f4-4d34-9e74-31401b34caf1   (Note that in this case, the User ID is sasadm. Remember that only administrators can export reports.)   Conveniently, the last column in the second record displays the URI to the package file. We can use the sas-admin CLI's transfer plug-in to display its contents.   /opt/sas/viya/home/bin/sas-admin transfer show-contents --uri /transfer/packages/85d85fac-f7f4-4d34-9e74-31401b34caf1   Name ContentUri ContentMediaType Retail Insights /reports/reports/cbf97b0a-457d-4b4f-8913-547e0cdf390c application/vnd.sas.report Samples /folders/folders/8af48321-d89c-4522-9557-079bce15edce application/vnd.sas.content.folder SAS Visual Analytics /folders/folders/f865778e-db27-4b4a-be3d-6322bd4ff81d application/vnd.sas.content.folder Products /folders/folders/1d8b18bf-853b-4a57-8600-c9bd37cf49ae application/vnd.sas.content.folder   In this example, the package contains the Retail Insights report and its corresponding SAS folder structure.   When importing packages, the transfer service will generate similar events that are also captured as audit records.   What about when exporting data to Excel from within a VA report? This will require successful reads to be turned on for the reportData application (reportData.action.read.state=all). We will then get the following records.   d81d2344-35ff-4bc8-af81-6ad957f37271 2019-10-02T03:34:32.592Z read success Ahmed reportData /reportData/results/310e8def-27a1-4f32-8676-05ce504d24b3_c1635e4d-618c-4968-9d9c-2d624fcb4279/exportFiles/dd431.xlsx ... 6778fbdb-2cce-4844-8fd1-beca87ce4cec 2019-10-02T03:34:32.445Z create success Ahmed reportData /reportData/jobs/c1635e4d-618c-4968-9d9c-2d624fcb4279   The read entry lists the Excel file, which is great, but notice anything that links it to the create entry? The URI contains a reference (after the underscore) to the create job entry.   Printing   Printing in Visual Analytics or Report Viewer generates a PDF of the report which the user can then print. This action results in the following audit records, indicating the name of the PDF that was produced.   4ebe39db-dd57-4c20-a61e-506f70602a9e 2019-09-25T11:34:14.826Z resource read success Ahmed reportRenderer 10.96.10.154 /reportRenderer/reports/3BIEOARCKYIRJU4KKR67XKTXDQAU3MWA/Retail%20Insights%20on%2009-25-2019.pdf 5fe92d47-ff6f-4134-880e-234985589c6d 2019-09-25T11:34:14.755Z resource create success Ahmed reportRenderer 10.96.10.154 /reportRenderer/reports/3BIEOARCKYIRJU4KKR67XKTXDQAU3MWA   Other Client-side Actions   Many other user actions performed in SAS Visual Analytics are client-side, meaning they do not interact with other services in a meaningful way (in this context). As such, they do not currently generate audit records in a manner that can be surfaced appropriately in SAS Viya 3.x. Auditing capabilities are constantly being improved, though, particularly in later versions of SAS.  Conclusion   Many organizations perform auditing for regulatory compliance or to report on security and usability aspects of the platform. Viya goes some way to meet basic auditing requirements out of the box, and provides the flexibility to expand the auditing scope with relative ease - BUT, do not forget the potential performance implications, as they can be significant. Keep an eye out for further improvements in auditing functionality in future releases. In later posts, we will explore auditing other areas of the SAS Viya platform.   For more information, refer to the official documentation.   Thanks to Eric Bourn, Todd Braswell and Anant Patil for their invaluable contributions to this post.   Thank you for reading. I hope the information provided in this post has been helpful. Please leave a comment below to share your own experiences.     Find more articles from SAS Global Enablement and Learning here. ... View more

When deploying SAS Viya, the default behavior schedules both stateful and stateless workloads across any available node pools. Generally, this setup is ideal, as Kubernetes is tasked with efficiently allocating pods to the appropriate nodes. However, there may be some edge cases where stricter scheduling of workloads is required. For example, it may be desirable to segregate stateful and stateless workloads to better utilise resources across different node pools, such as in situations where solutions like SAS Visual Investigator (with pods that demand more storage or memory) are deployed across node pools of varying sizes. In this post, we'll talk about confining stateful workloads to stateful node pools, and stateless workloads exclusively to stateless node pools.   Ensuring Workload Separation   The primary challenge lies in controlling where different types of pods are scheduled. By default, both StatefulSet (stateful) and Deployment (stateless) pods tolerate both stateful and stateless node classes. This flexibility can sometimes result in stateful pods being scheduled on stateless nodes, which might not have appropriate resource allocations, and could potentially lead to throttling or pod eviction during high activity from stateless pods.   Two approaches can be taken to address this challenge.   Setting up required node affinity   Node affinity allows us to define rules that specify where pods should be scheduled, based on the labels applied to nodes. This approach offers more granularity and flexibility, allowing for strict scheduling of workloads on particular nodes. In this case, we can switch the affinity rules from 'preferred' to 'required', as covered by my colleague Raphaël Poumarede in his post.   Begin by labeling the nodes to distinguish between stateful and stateless node pools.   kubectl label nodes workload.sas.com/class=stateful kubectl label nodes workload.sas.com/class=stateless   Add the following node affinity rules to StatefulSets to ensure these pods are scheduled only on stateful nodes.   spec: template: spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: workload.sas.com/class operator: In values: - stateful   Similarly, ensure that stateless pods (Deployments) are scheduled only on stateless nodes by modifying the YAML templates as follows:   spec: template: spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: workload.sas.com/class operator: In values: - stateless   Rebuild and redeploy to apply these changes.   Modifying tolerations   Tolerations work with taints applied to nodes, determining which pods are allowed to run on nodes with specific taints. By default, stateful and stateless pods tolerate both node classes. Modifying tolerations provides an alternative, simpler method of enforcing workload placement.   We can modify (remove) the tolerations in the YAML files that define the pods to achieve segregation of stateless and stateful workloads.   Here’s a sample patch to remove the stateful toleration from all Deployment resources to confine stateless pods to nodes optimised for stateless workloads.   --- apiVersion: builtin kind: PatchTransformer metadata: name: remove-stateful-toleration patch: |- - op: remove path: /spec/template/spec/tolerations value: - effect: NoSchedule key: workload.sas.com/class operator: Equal value: stateful target: group: apps kind: Deployment version: v1   An update to kustomization.yaml is also required:   ... transformers: - remove-stateful-toleration.yaml # Path to your patch file   Once applied, this patch will ensure that deployment resources are no longer scheduled on stateful nodes, improving the separation of workloads and enforce the changes across the environment.   Choosing the right approach   Both approaches outlined here offer effective methods for ensuring that stateful and stateless workloads are scheduled on the appropriate node pools in SAS Viya deployments. Other types workloads can also be segregated as desired. Note, however, that the default behaviour of scheduling across any available node pool is usually perfectly appropriate.   Required affinity provides more flexibility and control to enforce scheduling rules based on node labels, which can be easily adjusted if new node pools are added or removed. There is, however, an added administration overhead of maintaining node labels.   Modifying tolerations offers a simpler method for restricting pod placement by removing tolerations for nodes to repel unsuitable pods, but there is somewhat less flexibility, such as in handling changes to node pools.   Either method (or a combination of both methods) can be used to ensure stricter workload distribution in your SAS Viya environment with pods more efficiently distributed across node pools.   My thanks to Raphaël Poumarede, Harshit Soni and Henrique Lima for their contributions.       Find more articles from SAS Global Enablement and Learning here. ... View more

Auditing capabilities available in SAS Viya are essential for understanding user behavior and ensuring that resources are used appropriately. In this post, we'll demonstrate a method of tracking and viewing user access to SAS Visual Analytics reports using audit data in SAS Viya.   In SAS Viya, the Audit service is responsible for the capture and storage of records relating to user activity. By default, the Audit service does not capture successful 'read' events, so to capture information about who accessed a report, a simple change is first required to the sas.audit.record configuration instance in SAS Environment Manager's Configuration area.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   The audit.recording.level property can be adjusted to specify which events to capture. To enable the capture of successful 'reads', change the audit.record.level to a setting of High.     Note, though, that this can significantly increase the volume of data that is captured and stored. Your mileage may vary, but it’s important to balance capturing detailed information with managing storage requirements. One option is to define the service.list property, where you can specify a comma-separated list of application/service names for which successful 'read' events will generate audit records. It might also be prudent to adjust the retention periods for audit records.   Once that's done, the next time a user accesses a report, an audit record is created. In fact, many audit records are created. Suppose a user has viewed the sample Warranty Analysis report. The sas-viya --output text audit list --details command will return something like:   ID Time Stamp Type Action State Description User ID Application Remote Address Administrative Action URI 312c1b34-59b8-4fd9-97c8-016f9cb2ded2 2024-05-23T06:17:02.788Z resource read success Ahmed reports 127.0.0.1 true /Products/SAS Visual Analytics/Samples/Retail Insights.report 31d36d6b-6867-4840-b044-80b09a76c043 2024-05-23T06:17:02.69Z resource read success geladm reports 127.0.0.1 true /Products/SAS Visual Analytics/Samples/Warranty Analysis.report ceb2fadb-0bf2-4720-8e41-1eebb69a42c1 2024-05-23T06:17:02.596Z resource read success geladm reports 127.0.0.1 true /Products/SAS Visual Analytics/Samples/Water Consumption and Monitoring.report 4174fd1e-e0c9-498d-9394-a7ba955d58b1 2024-05-23T06:17:02.512Z resource read success geladm reports 127.0.0.1 true /Products/SAS Visual Analytics/Samples/Warranty Analysis.report 2841747c-fcdc-42b3-97cb-38bbcacd2bbd 2024-05-23T06:17:02.415Z resource read success geladm reports 127.0.0.1 true /Products/SAS Visual Analytics/Samples/Retail Insights.report c8e2e71c-dc49-4acf-bef4-42f57f27d1d1 2024-05-23T06:17:02.33Z resource read success geladm reports 127.0.0.1 true /Products/SAS Visual Analytics/Samples/Warranty Analysis.report bd9a5747-3871-470c-8010-3cbfe4c87ef1 2024-05-23T06:17:02.22Z resource read success geladm reports 127.0.0.1 true /Products/SAS Visual Analytics/Samples/Warranty Analysis.report 0540029e-08fe-4e59-9383-14cd2e7e40e7 2024-05-23T06:17:02.104Z resource read success Ahmed reports 127.0.0.1 true /Products/SAS Visual Analytics/Samples/Warranty Analysis.report 1417b61c-657a-4015-87d8-1b77a15bcb69 2024-05-23T06:16:42.226Z resource read success Ahmed reports 10.42.1.39 false /Products/SAS Visual Analytics/Samples/Warranty Analysis.report   Note that not only does Warranty Analysis appear multiple times as a successful read event, but other reports that we didn't open are also listed. This is because a "read" action is more than just the act of opening a report in SAS Visual Analytics. For example, when report thumbnails are rendered, these are also registered as "read" actions for the user, even if that user never actually opened the reports at that time.   How then do we differentiate between "reads" when a user physically opens a report and these other "reads"? To do so, we can use the RemoteAddress field (which is shown when the --details flag is added to the CLI command). The report that uniquely identifies the actual user action of opening the report contains the the podIP of the sas-folders pod as the value for the RemoteAddress field. We can get that with the command: kubectl get pod -l app=sas-folders -o custom-columns=NAME:metadata.name,IP:status.podIP   In my lab environment, that matches the last record displayed in the output above. The CLI command used to query the audit data could be modified to include a filter such that RemoteAddress equals the IP address of your sas-folders pod. Note that this behaviour is slightly different to the behaviour in SAS Viya 3.x.   Auditing the usage and access of SAS Visual Analytics reports is a critical aspect of managing and securing your SAS Viya environment. By leveraging the detailed audit data available, you can effectively monitor report access and usage and swiftly address any issues that arise. Implementing robust auditing practices not only aids in compliance but also promotes accountability and transparency within your organization.   For more information on auditing in SAS Viya, refer to the official SAS documentation.     Find more articles from SAS Global Enablement and Learning here. ... View more

In the world of monitoring and observability, the ability to detect and respond to issues promptly is paramount. Alertmanager and Grafana, both deployed with SAS Viya Monitoring for Kubernetes, possess alerting capabilities. While both serve the purpose of alerting, they have distinct features and approaches that cater to different needs. In this blog post, we'll endeavour to compare Alertmanager and Grafana by their alerting functions and provide some points to consider for when you might want to use one over the other.   Alertmanager's key features   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   Alertmanager is an open-source alerting tool that works seamlessly with Prometheus for a comprehensive metric-based monitoring and alerting toolkit. It is designed to handle alerts sent by client applications such as Prometheus and then manage these alerts by grouping, deduplicating, and routing them to various integrations like email, PagerDuty, Slack, and Microsoft Teams. Both are deployed together with SAS Viya Monitoring for Kubernetes. Since Alertmanager is tightly integrated with Prometheus, it leverages PromQL, Prometheus' powerful query language for defining alerting rules. This makes it easy to create sophisticated alerting conditions based on Prometheus metrics.   Alerts are created as PrometheusRules, and evaluated against metric data stored in Prometheus. Rules use YAML; easy to use, edit, store and deploy. Alertmanager allows users to define complex alert routing and inhibition rules based on labels attached to alerts. This flexibility enables fine-grained control over which alerts are sent to which receivers. For more details on how the routing process works, check out thisearlier post. Alertmanager provides a silence mechanism that allows users to temporarily mute specific alerts or entire groups of alerts.; useful when dealing with issues and addressing underlying causes. Alerts can also be inhibited in advance during planned outages, such as during maintenance activities. Alertmanager can be deployed in a highly available configuration to ensure reliability. It supports clustering and replication for fault tolerance.   Grafana's key features     Grafana is a popular open-source platform for monitoring and observability that offers a wide range of features, including visualization, dashboarding, and alerting. While Grafana is primarily known for its visualization capabilities, it also provides robust alerting functions. Older versions of Grafana required alerts to be created on dashboards, which was quite restrictive. While this is still possible, the Alerting facility that is now included offers much more comprehensive and flexible capabilities.   Grafana allows users to define alerting rules visually using its intuitive web interface. Users can set thresholds, conditions, and notification channels for triggering alerts directly in the UI (no need to write YAML or run kubectl commands). Grafana's alerting feature seamlessly integrates with its visualization capabilities. Users can create alerts directly from dashboards and visualize alerting states alongside other metrics. Grafana supports a variety of data sources; not just Prometheus. This flexibility enables users to create alerts based on data from different monitoring systems. Useful if other existing tools are already in use. Grafana keeps a history of triggered alerts, making it easy to track the lifecycle of alerts over time. Additionally, users can annotate graphs with alert events for better context.   Comparison   Authentication   Grafana requires users to log in with a username and password. Alertmanager has no such authentication built in; if the URL is accessible, anyone can use it. There are other ways to 'control' access to Alertmanager, but they need careful planning and consideration. Here is one example. Certainly, none are as convenient as Grafana's simple login prompt.   Ease of Use   Grafana's web interface provides an intuitive way to create and manage alerts (visually), making it more accessible to users who are not familiar with Prometheus' query language. On the other hand, Alertmanager's configuration may require more expertise due to its reliance on Prometheus' query language.     Default alerts   Prometheus includes a number of pre-defined alerts for Kubernetes applications when SAS Viya Monitoring for Kubernetes is deployed. Very useful for alerting on generally common issues, and no additional setup is required beyond defining notification channels for them. Grafana conveniently also displays these alerts, but interacting with them (e.g. silencing them when they fire) is a little less intuitive for these than it is for Grafana-managed alerts.     Integration   Both AlertManager and Grafana offer integration with various notification channels such as email, Slack, and PagerDuty. However, Grafana's integration capabilities extend beyond alerting to include visualization and dashboarding (Grafana is first a monitoring tool after all). There are also options for integrating the two together; for instance, to send Grafana-managed alerts to Alertmanager when they begin firing.   Scalability   Alertmanager can easily be scaled horizontally for high availability to handle large volumes of alerts; in fact, it's designed to be clustered. Grafana's scalability depends on the underlying data source it's connected to, but it can also scaled with a bit of effort. With SAS Viya Monitoring for Kubernetes, though, both are deployed as standalone instances by default.   Customisation   Alertmanager provides more fine-grained control over alert routing and inhibition rules, making it suitable for complex alerting workflows. It also offers options to customise nearly every aspect of the alert notifications. Grafana, on the other hand, offers greater flexibility in terms of visualization and dashboard customization, which Alertmanager lacks.   Conclusion   Both AlertManager and Grafana are powerful tools for alerting in the context of monitoring and observability. The choice between the two (or both!) depends on factors such as familiarity with Prometheus, integration and security requirements, and needs for advanced alerting features. Grafana may generally be considered more convenient for administrators with basic alerting requirements or whom are new to observability tools, whereas the expertise required to best leverage Alertmanager's powerful capabilities may be better suited to organisations that already use and have experience with it and related tools (e.g. Prometheus). Ultimately, organisations should evaluate their specific use cases and requirements to determine which tool best fits their needs. ... View more

The event-driven architecture in SAS Viya 3.5 (and 3.4) offers the capability for effective auditing of system resources, report activity, data access, and more. In this post, we’ll revisit the options available for auditing user access and highlight important caveats and considerations. This revised version of an earlier article has been updated to account for some changes in the underlying processes to provide a clearer picture.   CAUTION: Important considerations   Before adopting the approach to auditing user sessions documented in this post, beware these important cautions about using the EV Datamart data captured by the SAS Operations Infrastructure for this purpose: The data is not originally intended as 'audit' and it does not contain data integrity and security features that true audit data should. The data is derived from different data sources in which information is recorded differently, and matching user sessions with this data can be imprecise. It is not suitable for real-time or near-real-time purposes due to (potentially significant) latency in data being written to disk. Matching session activity like logons and logoffs using the session ID may not always work as expected. While the session ID is generally unique, it is not guaranteed to be unique. There are a small number of situations and edge-cases where a given session ID might be reused. SAS Viya 3.5 versions released in December 2023 (and later) address an issue where the session signature was not recorded for a login event, resulting in errors with the AUTHENTICATIONS table data. The information outlined in this post is a potential workaround for simplifying the process for auditing user session, but the caveats listed above must be noted as they present some potential challenges with this approach. Proceed with caution.   Information about logins and user sessions is captured by the Audit service in the AUDIT.security_audit tables in the SAS Infrastructure Data Server. The default behaviour, as defined in the security type properties in the sas.audit.record configuration instance, is to capture read failures only on security actions.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.    When trying to audit user login attempts, the login event provides information about (successful and failed) login attempts coming from the SASLogon application.   Let’s look at an example. Below, we use the sas-admin CLI’s audit plugin to see that user Ahmed logged in at 07:31 AM on the 25th of October (note that the timestamp is in UTC in PostgreSQL, but is converted to local time when a SAS Operations Infrastructure ETL job loads it to the AUDIT table in CAS).   /opt/sas/viya/home/bin/sas-admin --output text audit list --user-id Ahmed --sort-by ~timeStamp   ID Time Stamp Action State User ID Application URI c72c4533-886d-493b-9ef2-1dcff4ff4674 2019-10-25T07:57:54.776Z SessionDestroyed success Ahmed SASLogon bb748c54-6cb3-4edf-adbe-fce516c5786c 2019-10-25T07:31:21.014Z update success Ahmed reportImages /reportImages/jobs/4d7a64d8-1839-4d5b-a562-b140cdfc1f22/state caf1264d-4d37-444b-af45-4e339edc5dc6 2019-10-25T07:31:20.999Z create success Ahmed reportRenderer /reportRenderer/images 713e18ac-34db-4115-92aa-35f7662b0909 2019-10-25T07:31:20.649Z read success Ahmed reportData /reportData/commons/settings c5df6523-eb4f-42fe-b89b-fbdcc73d1c56 2019-10-25T07:31:20.298Z create success Ahmed reportPackages /reportPackages/jobs/b7325111-3f5a-4659-84bf-d1e59bdfbc74 e8578d9f-73b1-4dea-a94e-048a9eb9a72d 2019-10-25T07:31:20.030Z create success Ahmed reportData /reportData/jobs/61c55ce8-2b0a-4989-a14f-e1a5395b3b86 5aec043f-c2f8-46b6-a758-bc040abf1f69 2019-10-25T07:31:17.990Z create success Ahmed reportData /reportData/metadataSupplements fc7c64d8-bf75-40f5-a851-4afa5dc29beb 2019-10-25T07:31:16.141Z create success Ahmed reportImages /reportImages/jobs/4d7a64d8-1839-4d5b-a562-b140cdfc1f22 f9ad69f9-22b8-447b-82c2-45ebeb462409 2019-10-25T07:31:14.523Z read success Ahmed reportData /reportData/commons/settings 5dc96422-833c-496c-bd65-64659faf76d0 2019-10-25T07:31:08.641Z read failure Ahmed preferences /preferences/preferences/@currentUser/FormatLocale.DefaultLocale 5da5acf5-4d6b-47b3-9da5-c30ea7c3d946 2019-10-25T07:31:08.096Z read failure Ahmed preferences /preferences/preferences/@currentUser/_hc_xx_ 4a4c470f-afe9-469c-af3a-3fdb6d5f951e 2019-10-25T07:31:07.082Z read failure Ahmed preferences /preferences/preferences/@currentUser/FormatLocale.DefaultLocale b66f6eed-8ae8-47a3-8d3a-60e23df5cec9 2019-10-25T07:31:05.463Z login success Ahmed SASLogon   When a user logs off gracefully, a SessionDestroyed action is registered indicating the termination of the session as shown above.   If a user closes the browser instead of logging off, the SessionDestroyed does not get created in the audit records until the session times out. Session timeout duration is configurable.Set Time-out Interval for SAS Viya Web Applications in SAS® Viya® 3.5 Administration: Configuration ....   Some applications generate their own audit data when a user logs in. For example, SASStudio V creates a Compute server session when a user logs in, and an audit record similar to the following is captured.     /opt/sas/viya/home/bin/sas-admin audit list --user-id Ahmed --sort-by ~timeStamp --details --application compsrv   ID Time Stamp Type Action State Description User ID Application Remote Address URI 1967127d-87da-4656-922f-ed3b3bf1733d 2019-10-23T11:47:42.122Z resource start success 92ce6360-b23c-485b-94dc-48e5bd681bdd:https://intviya02.race.sas.com:38934/compute/servers/92ce6360-b23c-485b-94dc-48e5bd681bdd Ahmed compsrv   When the compute session is terminated, an audit record is captured indicating the deletion of the session (with the Description field indicating to the ID of the session).   /opt/sas/viya/home/bin/sas-admin audit list --user-id Ahmed --sort-by ~timeStamp --details --application compute   ID Time Stamp Type Action State Description User ID Application Remote Address URI 5ebd10ca-b4e1-4e7e-b71d-2874bfa672ca 2019-10-23T07:50:36.257Z resource delete success Deleted compute server 92ce6360-b23c-485b-94dc-48e5bd681bdd Ahmed compute /compute/servers/92ce6360-b23c-485b-94dc-48e5bd681bdd c9a1fc4a-48ea-4afa-a456-45e890571e42 2019-10-23T07:50:36.132Z resource delete success Deleted compute server session 92ce6360-b23c-485b-94dc-48e5bd681bdd-ses0000 Ahmed compute /compute/sessions/92ce6360-b23c-485b-94dc-48e5bd681bdd-ses0000   User sessions are assigned a session signature when they are established. The session signature (session_sig) property links the login event with the corresponding SessionDestroyed event. However, there are some important considerations.   First, refer to the caveats listed at the beginning of this post about potential issues with relying on the session signature for this purpose.   Second, the session signature is not visible when surfacing audit records using the CLI. This information is recorded in TSV files in the EV Datamart directory (/opt/sas/viya/config/var/lib/evmsvrops/evdm/application/security) on the Viya server. When a session is started:   2 34dca2cc-fcc9-4b6d-8c72-32d2958c9320 security application/vnd.sas.event.security 2019-10-15T06:21:18.973000+00:00 sasadm sas-deployment-id:dml5YQ==,sas-event-source:U0FTTG9nb24=,session_sig:NTNlZmNlZGE= security action:bG9naW4=,actionState:U1VDQ0VTUw==   Note the session_sig value. When the session ends, another record will be generated in the same file, referring to the original signature (orig_session_sig) for that session.   2 b41e8882-192d-4657-8fee-4cb04a96abda security application/vnd.sas.event.security 2019-10-15T07:02:30.282000+00:00 sasadm orig_session_sig:NTNlZmNlZGE=,sas-deployment-id:dml5YQ==,sas-event-source:U0FTTG9nb24=,session_sig:NDk0MGZmNw== security action:U2Vzc2lvbkRlc3Ryb3llZA==,actionState:U1VDQ0VTUw==   Fortunately, the legwork for extracting this information has already been done. Session information is extrapolated from these files and stored in the authentications subject table (opt/sas/viya/config/var/lib/evmsvrops/evdm/subjects/authentications.sas7bdat). Information is captured in this table for metering purposes, a secondary effect is that it provides a really nice way (with the caveats listed above!) to audit user session information. In the example below, several user sessions are listed, each with a datetime stamp of the login action, the session signature, and the duration of the session (in seconds).   Note that user Sasha’s session has a duration of 0.00 seconds. This points to an active user session (but there may be exceptions - listed above -  due to latency), indicating that Sasha has not yet logged out (SessionDestroyed action has not occurred for the session with this signature).   A job runs every 5 minutes to update this table, and another job runs daily to clear records older than 10 days. Update: To change the number of days (e.g. from 10 to 14 days) for which to capture records, execute:   /opt/sas/viya/home/bin/ops-dm-admin set EMI_DELETE_TSVZIP_DAYS=14   We can query this table to get some more insights about user sessions. Can you use it to figure out who was logged in at a certain time, for example?   A basic SAS program can be submitted to help answer that question.   /* Specify query time below in datetime. format */ %let qrytime='15OCT19:08:39:18'dt; libname evdm '/opt/sas/viya/config/var/lib/evmsvrops/evdm/subjects' access=readonly; data activesessions; set evdm.authentications; endtime=datetime+duration; format datetime datetime.; format endtime datetime.; where &qrytime between datetime and &endtime or (&qrytime >= datetime and (duration=0 and session_sig ne ''); run; proc print data=activesessions; run;   Note that this code sample is for demonstration purposes only, and there may be more efficient or more accurate methods to extract the required user session information.   Thanks to my colleagues Greg Smith, Carl Sommer, Fred Forst and Mike Roda for their contributions to this and the original version of this article, and thank you for reading! ... View more

PostgreSQL 15 is available in the November 2023 release of SAS Viya 3.5. SAS strongly recommends that customers upgrade their deployed PostgreSQL instance to version 15. Customers running SAS Viya 3.5 should check to see if they already have version 15 deployed, and if not, should consider upgrading in order to stay comfortably within PostgreSQL's support policy and to stay current with security fixes and patches. In this post, we will demonstrate how to check what version you are running in your environment, and highlight the steps you must perform to upgrade your deployed version of PostgreSQL to version 15.   If you are running a version of SAS Viya older than Viya 3.5, you must first upgrade your environment to SAS Viya 3.5 if you want to obtain PostgreSQL 15. Specifically, you should upgrade to at least Ship Event 23w44, which is the earliest release of SAS Viya 3.5 that contains PostgreSQL 15. This means that to get PostgreSQL 15, you must obtain a new order from SAS and then perform an upgrade. Any new deployments of SAS Viya 3.5 from 23w44 or later will deploy PostgreSQL version 15 by default. Similarly, if you are running a version of Viya 3.5 earlier than Ship Event 23w44, you need to upgrade to at least that release to get PostgreSQL 15. The simplest way to verify the version you are running is to directly query the version of PostgreSQL deployed in your environment by doing the following:   Determine the deployment target(s) for PostgreSQL in your environment In you inventory.ini file, look in the [sasdatasvrc] and [pgpoolc] (if applicable) host groups to identify the target machines, and then look in the [host_definitions] at the beginning of the file to view host information for those machines. Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   Check if the PostgreSQL 15 binaries have been deployed If SAS Viya is deployed on Linux, run the following command on the target machines: ls /opt/sas/viya/home/postgresql15 If not found, you do not yet have PostgreSQL 15 deployed.   Or if deployed on Windows, run the following in Powershell: dir "C:\Program Files\SAS\Viya\lib\postgresql15\bin"   If not found, you have not yet deployed PostgreSQL 15.   In these cases, you must contact SAS to obtain a new SAS Viya 3.5 software order from Ship Event 23w44 or later to upgrade to PostgreSQL 15. Once you have the new order, upgrade your SAS Viya deployment by following the documented instructions, remembering to perform the all-important pre-upgrade and post-upgrade steps for PostgreSQL. If you do have the PostgreSQL 15 binaries deployed, continue to the next step.   Verify whether your SAS Viya deployment is actually running the deployed PostgreSQL 15.   It is possible that your environment has PostgreSQL 15 deployed, but post-upgrade steps to configure it haven’t been performed. As an additional check, verify whether your SAS Viya deployment is actually running the deployed PostgreSQL 15.   On Linux, run the following as sudo on one of the target machines identified earlier: sudo cat /opt/sas/viya/config/data/sasdatasvrc/postgres/node0/PG_VERSION On Windows, run the following in Powershell:   cat C:\ProgramData\SAS\Viya\data\sasdatasvrc\postgres\node0\PG_VERSION A value of 15 listed in the PG_VERSION file indicates that PostgreSQL 15 is running and in use by SAS Viya. You can stop reading here, because all is as it should be!   However, if the listed version is not 15, some further action is required. We need to check to see if the latest PostgreSQL and related RPMs/MSIs are installed. On Linux, run the following to check the deployed RPM versions for PostgreSQL components:    cd /etc/init.d rpm -qa | egrep 'sasdata|sas-postgres|pool'   The versions displayed might resemble the below (the latter part of the values displayed for each RPM is irrelevant and has been omitted):   sas-sasdatasvrc-0.8.1-XXXXXXXXX.XXXXXXXX sas-postgresql-secure-libs-15-15.4-XXXXXXXXX.XXXXXXXX sas-pgpoolc-0.7.0-XXXXXXXXX.XXXXXXXX sas-postgresql-secure-15-15.4-XXXXXXXXX.XXXXXXXX sas-pgpool-II-44-4.4.4-XXXXXXXXX.XXXXXXXX If the versions shown above are displayed, this indicates you have deployed PostgreSQL 15 but not configured it yet. Some additional steps are required to reconfigure PostgreSQL and pgpool and complete the upgrade to version 15. Review the post-upgrade steps and completed all documented tasks.   If you do not have those versions, contact SAS to obtain a new SAS Viya software order from Ship Event 23w44 and upgrade your SAS Viya deployment in order to obtain PostgreSQL 15 Similarly, on Windows deployments, run the following in Powershell to check the MSI versions:    dir /powershell-deployment\Downloads   If sas-datasvrc15-15.4.0.0.msi   is listed, you have PostgreSQL15 deployed but not configured yet. Again, review the post-upgrade steps and completed all documented tasks. If you do not have that version, contact SAS to obtain a new SAS Viya software order from Ship Event 23w44 and upgrade your SAS Viya deployment in order to obtain PostgreSQL 15. A couple of additional points to consider:   Perform a full backup after updating your SAS Viya deployment After upgrading PostgreSQL, do not restore backups that were taken in older versions of PostgreSQL. If you have upgraded to PostgreSQL version 15, and you wish to then migrate to SAS Viya 4, you must move to a supported target version of SAS Viya 4 using an external instance of PostgreSQL 15.  Thank you for reading. My thanks also to my colleagues Cindy Taylor, Joseph Cho and Fred Perry for their contributions.        ... View more

Staying up-to-date with SAS Viya software is more important than ever. Updating regularly will give you the latest features and fixes, but also ensures that your deployment doesn't fall out of Standard Support. Before you set sail on your voyage to update to new SAS Viya version, it's essential to prepare properly to ensure a safe and smooth transition. In this article, we'll explore the important pre-upgrade tasks that will help reduce the risk of unforeseen issues and ensure you are set up for a successful update.   Pre-update Checklist   The SAS Viya Platform Administration documentation lists the SAS Viya platform versions that are in Standard Support and the recommended update paths to the latest version. In fact, the task of checking the support status of your deployment is the very first item in the Pre-update checklist of essential tasks that SAS now provides, which should be consulted before each update. If skipping versions or switching cadence as part of your update, take care to correctly identify interim versions which may be relevant; you will need to consult the Deployment Notes and execute the documented Before Deployment Commands for these.   In addition to checking your SAS Viya software support status, the checklist includes a task to verify your deployed Kubernetes version. Carefully read the documentation to see if you also need to upgrade to a supported version of Kubernetes or Kustomize.    Additionally, if your deployment includes it, consider updating the SAS Deployment Operator with each SAS Viya software update. Updates are backwards compatible, so running the latest version is a good practice. Note that if you have the SAS Deployment Operator deployed, you must use it to perform the SAS Viya update. If you used the sas-orchestration deploy method for the initial install, you must use that. It is not possible to mix and match deployment methods for update tasks.   A good practice is to run an inventory scan report before and after any major platform changes, including software updates. The two scans can be compared to verify that your content was not affected/changed during the update.   The inventory scan should also be run prior to a backup (and if required, after a restore). Pre-update backups should include an ad-hoc SAS Deployment Backup, backups of the $deploy directory and kustomization file, and if applicable, the SASDeployment CR file.   Next on the checklist is Deployment Assets. These should be retrieved for the target version you are moving to. Once downloaded, take a look at the README.md files for products you have deployed - this step is particularly important if you are adding new software as part of an upgrade, as there may be additional kustomizations or other manual steps that need to be performed for those new products. Note that assets automatically include the latest available license.   There are additional tasks in the checklist for multi-tenant environments and deployments that include SingleStore or multiple CAS servers. Be sure to work diligently through the list to ensure all relevant tasks are appropriately addressed.   Additional Tasks   Updates should be carefully co-ordinated with relevant technical resources to complete pre-requisite tasks. You may need to liaise and work with your Kubernetes administrator, security teams, change managers, end users, etc. to ensure you are ready for the update to begin. For example, you may need resources assigned to perform supplementary tasks ahead of time, such as:   providing sufficient access to enable you to perform the software update upgrading Kubernetes software (if required) verifying that the cluster has sufficient available resources to enable duplicates of each pod to run temporarily during the update notifying users of potential disruption updating the mirror registry, if applicable retrieving the output of the update checker job (bonus points for automating this)   Each organisation may have unique considerations, so seek further guidance from SAS Technical Support if necessary. With a well-executed pre-upgrade plan, you will avoid pitfalls and minimise disruptions on your way to the latest version of SAS Viya.     Find more articles from SAS Global Enablement and Learning here. ... View more

Grafana provides comprehensive insights into your SAS Viya deployment's health and performance using metrics collected by Prometheus. New capabilites were recently added to Grafana to improve its alerting capabilities. In this post, we will walk you through the process of configuring alerting for your SAS Viya deployment in Grafana.   Grafana requires Prometheus or another compatible data source set up to gather metrics from SAS Viya components. This is the default when SAS Viya Monitoring for Kubernetes is deployed.   The Grafana configuration is stored in a configMap (v4m-grafana) in the monitoring namespace. The config stored here includes things like your SMTP server host and port. If Grafana is already deployed, running a set of commands like the below will toggle on email connectivity and specify your mail server details in the configMap to allow Grafana to send alert notifications via email.   # get configmap in .yaml kubectl --namespace monitoring get configmap v4m-grafana --output=yaml > /tmp/v4m-grafana-configmap.yaml # update yaml file to enable smtp and enter mail server host:port sed -r -i "/kind: ConfigMap/ i\ \ \ \ [smtp] \\ \ \ \ enabled = true \\ \ \ \ \ host = mailserver.gelcorp.sas.com:25" /tmp/v4m-grafana-configmap.yaml # apply updated yaml kubectl -n monitoring replace --filename=/tmp/v4m-grafana-configmap.yaml # restart grafana kubectl -n ${_monitoringNS} delete po -l app.kubernetes.io/name=grafana   If Grafana isn't deployed, it's possible to specify these config parameters (and others) pre-deployment in the user.env file provided with SAS Viya Monitoring for Kubernetes.   Before actually getting to creating alerts, you must configure the notification channels to receive alerts. First, log on to Grafana and navigate to the "Alerting" section by clicking on the bell icon in the left sidebar.   Note that the main "Alert rules" page displays any other alerts configured in Prometheus. While there are limited options for interacting with these alerts, having the ability to view all alerts in one interface is very convenient. The best part is that when SAS Viya Monitoring for Kubernetes deploys Grafana, it is already configured to use Prometheus as a data source for metrics, so these alerts appear automatically without any additional config.   Click on the "Contact points" tab. This is where we configure our channels. A default one, grafana-default-email, is pre-defined for us. Click the pencil icon to open the edit page, where you can select the contact point type (Email) and then specify the destination address. Click the Test button after doing so to make sure everything is set up correctly for the recipient to receive the notification.     Save the changes.   Now head back to the Alert rules tab. The new alert we create in this page will be a Grafana managed alert (i.e. not a PrometheusRule). Click on "+ New alert rule" under Grafana alerts to begin adding the alert. Define the alert conditions based on SAS Viya metrics collected by Prometheus (using either the builder or code methods). Grafana's interface for creating alerts provides a rich set of options for controlling all aspects of your alert condition, and includes the best features of the Prometheus Expression Browser and then some. Specify or select the appropriate metrics, choose how they are filtered and evaluated, and then set the threshold values and evaluation frequency. For example, you could set a rule to trigger when memory usage exceeds a certain value for a specified duration.   Craft an informative rule name and message as well as other descriptive attributes that clearly indicate the issue. You will also be required to select (or create) a folder and group for your rule. In the "Notifications" section, select the key-value labels that will connect the alert to contact points (the equivalent of Alertmanager's routing function).   Save the alert. When the evaluation period lapses, and the alert condition is met, the alert will begin firing and a notification will be sent.         Adjust the threshold values, duration, or other settings as needed based on your alerting strategy and requirements.   We'll look at next steps and additional options in a future post. Stay tuned.   Thank you for reading.   ... View more

As part of ongoing efforts to enhance SAS Viya administration capabilities, we will introduce in this blog post a new program within the pyviyatools repository. getactivityrecords.py is a Python script designed to retrieve and analyze activity records from the SAS Viya environment. These activity records are a kind of audit record that provide insights into the operations and usage patterns of your SAS Viya deployment, enabling administrators and audit compliance folks to better understand at a high-level how the Viya deployment is being used by end users.   The concept of activities has been introduced within SAS Viya's audit service to provide a way to inject higher-level business logic into the auditing process. Traditionally, auditing focuses on low-level technical details, providing limited visibility into the actual usage patterns of different endpoints, making it somewhat challenging to derive meaningful insights. With the introduction of activities, the auditing framework gains a new, higher-level perspective to make auditing more intuitive and understandable. This improved understanding allows for better analysis, troubleshooting, and decision-making based on the audit data.   The audit service routinely loads activity data, which is stored in PostgreSQL, into the SystemData.AUDIT_ACTIVITIES table in CAS for reporting. The User Activity VA report provides a way for users to visualise audit data, but there is not yet a way to do the same for activity data, nor to view the raw activity records themselves.   Enter getactivityrecords.py, the latest addition to the growing pyviyatools repository.   To get started with getactivityrecords.py, you can simply clone the pyviyatools GitHub repository and install the necessary dependencies. When it is run, getactivityrecords.py leverages the SAS Viya REST API to retrieve activity records from the SAS Infrastructure Data server. It also allows you to specify the time range and filters to narrow down the scope of the records you want to retrieve. These records can not yet be surfaced from the sas-viya CLI's audit plug-in, which instead retrieves only the 'regular' audit records.   The program supports multiple output formats, including CSV (default), JSON, and Pandas DataFrames. This flexibility allows you to seamlessly integrate the retrieved activity records into your preferred reporting tools or visualization platforms. Leveraging scheduling tools like cron, you can automate the execution of getactivityrecords.py at regular intervals, enabling the continuous capture of activities within the SAS Viya environment.   The simplest way to execute the program is to simply run it without any additional parameters: getactivityrecords.py     Output:   id ,type ,action ,administrativeAction ,state ,user ,application ,timeStamp ,remoteAddress "2e88b242-1e6b-4aae-bab0-5cdb740d5aee","resource","create","False","success","dagentsrv-gelcorp","identities","2023-06-13T09:15:34.471Z","10.42.1.13" "8d6a74dc-5219-41b4-aaf7-538b24a8e41a","resource","create","False","success","dagentsrv-gelcorp","identities","2023-06-13T09:15:35.877Z","10.42.1.13" "e40bf1ec-47db-4d95-89a0-aa7d3a5204a4","security","login","False","success","geladm","SASLogon","2023-06-13T09:21:54.678Z","127.0.0.1" "efd0af8e-6439-4a3e-97e9-fe50efa5a163","security","login","False","success","geladm","SASLogon","2023-06-13T09:21:55.61Z","127.0.0.1" "5394eeb4-a321-4a3a-bd2c-0b7a4cb01438","security","login","False","success","geladm","SASLogon","2023-06-13T09:22:02.093Z","127.0.0.1" "37e04c46-693f-4899-bb74-3a84a593f184","security","login","False","success","geladm","SASLogon","2023-06-13T09:22:03.771Z","127.0.0.1" "0d4d1a45-2b12-4399-89ba-57de5a6753c7","security","login","False","success","geladm","SASLogon","2023-06-13T09:22:04.449Z","127.0.0.1" "f2ce150d-613b-48c4-812f-7cb86ecad433","security","login","False","success","geladm","SASLogon","2023-06-13T09:22:05.314Z","127.0.0.1" "de62ccdc-1e42-403c-a047-3676812219cc","security","login","False","success","geladm","SASLogon","2023-06-13T09:22:12.019Z","127.0.0.1" "84f530bf-4f93-4b78-893e-e200bf0cba68","security","login","False","success","geladm","SASLogon","2023-06-19T02:53:41.367Z","127.0.0.1" "be704891-ac56-4646-bd89-18206597a4ab","security","SessionAuthenticationSuccess","False","success","geladm","SASLogon","2023-06-19T02:53:41.375Z","127.0.0.1" "fbad594a-8571-4e32-83b3-95204d90d930","security","AdministrativeAction","True","success","geladm","SASLogon","2023-06-19T02:53:48.372Z","127.0.0.1" "e8556c28-e72d-4f3f-b281-207f12f15f68","security","login","False","success","geladm","SASLogon","2023-06-19T03:22:09.338Z","127.0.0.1" "4591fdf7-25c6-4ea2-b705-ab60e89b5fbf","security","SessionDestroyed","True","success","geladm","SASLogon","2023-06-19T03:24:38.374Z","127.0.0.1" "c8005e77-0250-4287-bd45-5eda35238687","security","login","False","success","geladm","SASLogon","2023-06-19T04:28:44.194Z","127.0.0.1" "7a012678-70ae-424c-9088-c5890e9a2d7c","security","SessionAuthenticationSuccess","False","success","geladm","SASLogon","2023-06-19T04:28:44.198Z","127.0.0.1" "d85ebff4-74e4-4762-8b3c-3802a43a2504","security","AdministrativeAction","True","success","geladm","SASLogon","2023-06-19T04:28:45.906Z","127.0.0.1" "aa407dcb-50fb-4951-86ca-5e39c9db5989","security","login","False","success","Ahmed","SASLogon","2023-06-19T04:31:02.689Z","127.0.0.1" "15a68ff7-a6cc-4b18-a8df-be770269cf61","security","SessionAuthenticationSuccess","False","success","Ahmed","SASLogon","2023-06-19T04:31:02.691Z","127.0.0.1" "4c8f6b3b-32ca-4f00-b3fe-4d4a482624ad","security","SessionDestroyed","False","success","Ahmed","SASLogon","2023-06-19T04:37:00.375Z","127.0.0.1" "cf42e5fa-ea42-430d-b8a7-c095be8c9194","security","login","False","success","Delilah","SASLogon","2023-06-19T04:37:05.142Z","127.0.0.1" "17404791-57b0-4f62-9c1d-5c763ec4ceda","security","SessionAuthenticationSuccess","False","success","Delilah","SASLogon","2023-06-19T04:37:05.149Z","127.0.0.1" "50d7eb6e-89d9-42f3-817a-591dde1ca3af","security","SessionDestroyed","True","success","geladm","SASLogon","2023-06-19T04:59:38.474Z","127.0.0.1" "6a44bbf8-b0a3-4171-9e02-337364bdeb6e","security","login","False","success","Delilah","SASLogon","2023-06-19T05:22:14.95Z","unknown" "1af2d753-5235-4cdf-b754-bc5397d2661e","security","SessionDestroyed","False","success","Delilah","SASLogon","2023-06-19T05:32:38.52Z","127.0.0.1"   As the sample output above shows, activity records are currently capturing predominantly security-type records. This is expected to grow in future releases to include records from more and more applications for capturing additional types of user actions.   You can also add additional flags for filtering the output. For example, you can narrow the results by user, application, action, or time range. Consult the built-in documentation to view all options: getactivityrecords.py -h   Output:   usage: getactivityrecords.py [-h] [-l LIMIT] [-t TYPE] [-a APPLICATION] [-c ACTION] [-d ADMIN_ACTION] [-s STATE] [-u USER] [-A AFTER] [-B BEFORE] [-S SORTBY] [-o {csv,json,simple,simplejson}] optional arguments: -h, --help show this help message and exit -l LIMIT, --limit LIMIT Maximum number of records to display -t TYPE, --type TYPE Filter by entry Type -a APPLICATION, --application APPLICATION Filter by entry Application -c ACTION, --action ACTION Filter by entry Action -d ADMIN_ACTION, --admin-action ADMIN_ACTION Filter by Administrative Action -s STATE, --state STATE Filter by entry State -u USER, --user USER Filter by Username -A AFTER, --after AFTER Filter entries that are created after the specified timestamp. For example: 2020-01-03 or 2020-01-03T18:15Z -B BEFORE, --before BEFORE Filter entries that are created before the specified timestamp. For example: 2020-01-03 or 2020-01-03T18:15Z -S SORTBY, --sortby SORTBY Sort the output ascending by this field -o {csv,json,simple,simplejson}, --output {csv,json,simple,simplejson} Output Style   Understanding how your SAS Viya environment is being used can help SAS Viya administrators to meet regulatory obligations, monitor and review security aspects, troubleshoot effectively and better align platform usage with broader business objectives. By leveraging the power of Python and the flexibility of SAS Viya's REST API, getactivityrecords.py enables the retrieval and analysis of activity records to provide valuable insights into user patterns for administrators and auditors.     Find more articles from SAS Global Enablement and Learning here. ... View more

In today's fast-paced world of infrastructure management, monitoring and alerting systems play a vital role in ensuring the health and performance of applications and services. An earlier post introduced the alerting capabilities provided by Prometheus for SAS Viya, which help administrators identify anomalies and respond to incidents proactively. This refresher article provides a high-level recap of what is available out of the box, what it can help with, and covers the process of creating new, custom alerts to enhance real-time monitoring.   Alerting is the process of triggering actions when defined conditions are met. In a SAS Viya deployment (with SAS Viya Monitoring for Kubernetes deployed), those conditions are based on metric information collected by Prometheus (about health and performance of your SAS Viya deployment) or log data collected by OpenSearch.   Alerts can help administrators proactively detect and predict potential anomalies with the deployment and/or infrastructure. When a defined alert condition or threshold is met, Prometheus lets the companion Alertmanager module know. Alertmanager can send notifications to a variety of channels, as well as take other actions automatically when an alert is triggered (such as creating a JIRA or ServiceNow ticket, or even autoscaling up/down). It also provides a facility for managing the lifecycle of triggered alerts (for example, to supress alerts while they’re being investigated).   So what kind of conditions can trigger alerts? When Prometheus is deployed, it includes a heap of generally useful alerts for common conditions out-of-the-box. These include alerts for scenarios like pods stuck in a CrashLoopBackOff state, StatefulSets not matching the expected number of replicas, Persistent Volumes filling up, or elevated CPU throttling. But administrators can also supplement these with their own bespoke alerts.   Before we get to the "how", let’s look at the type of metric information available in Prometheus. This can be done by submitting PromQL (the native, SQL-based querying language) expressions in the Prometheus Expression Browser web application. Conveniently, the Expression Browser includes a metrics explorer and has auto-complete, so you can begin by just selecting a metric and viewing the data available, and then adjusting your expression to filter for what you’re interested in seeing (and alerting on).   For example, assume we want to see memory usage for the sas-cas-control pod. If the word “memory” is typed into the query box, matching metrics are displayed and can be selected from the drop-down box. The container_memory_usage_bytes metric looks like a good match for our use case. A filter can be added by appending a curly brace to the query. We want to filter by pod, so when we start typing pod=" , auto-complete again lets us click on the desired pod name from a drop-down list.   In this example, our resulting expression is container_memory_usage_bytes{pod="sas-cas-control-6c9455dd78-g58fr"} . If we hit the Execute button, we get the metric data values (in bytes) for the query.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   Assume we want to create an alert for this metric. For instance, say we want an alert to be triggered when the sas-cas-control pod exceeds 80% of its memory limit, which we can find by running a kubectl describe on the pod (or for bonus points, try using the Expression Browser to find the appropriate metric and filters that will also give you the limit). Our expression would then become something like [memory usage] / [memory limit] as a percentage, which translates to PromQL as: (sum by (pod) (container_memory_usage_bytes{pod="sas-cas-control-6c9455dd78-g58fr"})) / (sum by (pod) (kube_pod_container_resource_limits{pod="sas-cas-control-6c9455dd78-g58fr",resource="memory"})) * 100   We can then add an operator to our expression to define the alert threshold - if we want to alert the trigger when the usage reaches 80% of the limit, the expression becomes: (sum by (pod) (container_memory_usage_bytes{pod="sas-cas-control-6c9455dd78-g58fr"})) / (sum by (pod) (kube_pod_container_resource_limits{pod="sas-cas-control-6c9455dd78-g58fr",resource="memory"})) * 100 > 80   The process for creating the alert is outlined in an earlier post, but to recap, requires a new YAML file which defines the alert in a PrometheusRule (custom resource) definition to be created and submitted with kubectl:     tee ~/PrometheusRule.yaml > /dev/null << EOF apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: labels: prometheus: prometheus-v4m-prometheus-0 role: alert-rules name: prometheus-viya-rules namespace: monitoring spec: groups: - name: custom-viya-alerts rules: - alert: CASControlMemoryApproachingLimit annotations: description: sas-cas-control container memory usage is approaching its limit. This can result in OOM errors and CAS sessions closing unexpectedly. summary: CAS Control high memory usage expr: (sum by (pod) (container_memory_usage_bytes{pod="sas-cas-control-6c9455dd78-g58fr"})) / (sum by (pod) (kube_pod_container_resource_limits{pod="sas-cas-control-6c9455dd78-g58fr",resource="memory"})) * 100 > 80 labels: severity: critical EOF kubectl -n monitoring create --filename ./PrometheusRule.yaml       Once created, the alert will be visible in the Prometheus UI's Alerts page:     If it when it triggers, it will turn red, and will also appear in the separate Alertmanager UI. When alert conditions are addressed/remediated, the alerts automatically return to a state of “Inactive” (not firing) without any intervention from an administrator. Administrators can "silence" firing alerts in the Alertmanager UI as a way of acknowledging them while they are being investigated.   Obviously, admins still need to specify what happens when the alert does trigger – for example, deciding who gets notified and how. This requires modifying the Alertmanager configuration. A good starting point is this post on alert routing. There are several earlier articles which outline how to configure Alertmanager to integrate with various third-party applications such as MS Teams, and Slack. . The command-line based amtool can also be very useful here.   Leveraging the powerful and flexible alerting capabilities provided by Prometheus in SAS Viya can enable organizations to stay ahead of critical situations, respond promptly to incidents, and optimize their operational efficiency.   For additional information, refer to the official Prometheus documentation and the SAS documentation on SAS Viya’s logging and monitoring. ... View more

Update to post: The open source configuration is available to SAS compute and batch sessions as of 2024.07, along with new, simplified documentation consolidating the instructions for configuring Python integration.   One of the key features of SAS Viya is its ability to integrate with different programming languages, including Python. SAS users can submit Python code for execution in SAS Viya with the PYTHON procedure; check out this blog post for an overview of what's possible. Some configuration changes are required to get set up, after which users can start running Python code from a SAS Compute session. That's great for submitting from SAS Studio, but what about in batch?   The necessary setup is covered nicely by Scott McCauley in his article. When the steps are completed, users can submit Python code or programs for execution in a SAS Compute pod (e.g. in a SAS Studio session).   However, a peek inside the file site-config/sas-open-source-config/python/python-transformer.yaml suggests that these configuration changes will only apply to the "sas-compute-job-config" PodTemplate; that's the Custom Resources that tells Kubernetes how to initialise a SAS Compute pod. Any new SAS Compute pod that gets launched will contain these changes to support Python code execution. However, things work a little differently with batch submission of code. The sas-viya CLI has a batch plugin that allows SAS programs (which can contain a PROC PYTHON statement to embed Python code) to be submitted from the command line. With this method, the submitting user must specify a batch context (as opposed to a compute context) to use. By default, Viya is deployed with two batch contexts; default and default-cmd. Both of these use the "sas-batch-pod-template" PodTemplate to initialise a batch session to execute the request. Unfortunately, this template will not contain the changes made to the compute template that are necessary to run Python or R.   We can see this difference by comparing the the compute PodTemplate with the batch PodTemplate.   Running kubectl describe podtemplate sas-compute-job-config shows all the Python configuration, such as the required volume mounts, is present:   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   Those are missing in the output shown by running kubectl describe podtemplate sas-batch-pod-template . This means that when we try to run a SAS program containing Python code, it will fail, since none of the required configuration will exist in the pod launched to run the code.   The solution, then, is to modify the batch PodTemplate so that batch sessions also contain the required open source configuration.  We can just copy and paste the lines that already exist in the transformer for patching the compute PodTemplate to also patch the batch PodTemplate.   The lines of interest are towards the end of python-transformer.yaml:       apiVersion: builtin kind: PatchTransformer metadata: name: compute-job-python-transformer patch: |- # Add python volume - op: add path: /template/spec/volumes/- value: name: python-volume persistentVolumeClaim: claimName: sas-pyconfig # Add mount path for python - op: add path: /template/spec/containers/0/volumeMounts/- value: name: python-volume mountPath: /opt/sas/viya/home/sas-pyconfig readOnly: true # Add python-config configMap - op: add path: /template/spec/containers/0/envFrom/- value: configMapRef: name: sas-open-source-config-python target: kind: PodTemplate name: sas-compute-job-config version: v1       After copy and pasting the above section, and then editing to target the batch PodTemplate, the end of your python-transformer.yaml should now also include:       ... --- apiVersion: builtin kind: PatchTransformer metadata: name: enable-python-batch-transformer patch: |- # Add python volume - op: add path: /template/spec/volumes/- value: name: python-volume persistentVolumeClaim: claimName: sas-pyconfig # Add mount path for python - op: add path: /template/spec/containers/0/volumeMounts/- value: name: python-volume mountPath: /opt/sas/viya/home/sas-pyconfig readOnly: true # Add python-config configMap - op: add path: /template/spec/containers/0/envFrom/- value: configMapRef: name: sas-open-source-config-python target: kind: PodTemplate name: sas-batch-pod-template version: v1       If you've done all the earlier pre-req steps, your kustomization.yaml file won't require any further changes, so you can just build and apply (or deploy your sasdeployment custom resource if you have the SAS Deployment Operator configured).   Now let's verify the changes. When deployment completes, run kubectl describe podtemplate sas-batch-pod-template|grep -A5 -B5 sas-pyconfig to check that the batch PodTemplate has been updated:       k:{"mountPath":"/opt/sas/viya/home/commonfiles"}: .: f:mountPath: f:name: f:readOnly: k:{"mountPath":"/opt/sas/viya/home/sas-pyconfig"}: .: f:mountPath: f:name: f:readOnly: k:{"mountPath":"/sashelp"}: -- Mount Path: /opt/sas/viya/config/etc/SASSecurityCertificateFramework/private Name: security Sub Path: private Mount Path: /gelcontent Name: sas-viya-gelcorp-volume Mount Path: /opt/sas/viya/home/sas-pyconfig Name: python-volume Read Only: true Dns Policy: ClusterFirst Image Pull Secrets: Name: sas-image-pull-secrets-k95ckd46h8 -- Nfs: Path: /shared/gelcontent Server: pdcesx02072.race.sas.com Name: python-volume Persistent Volume Claim: Claim Name: sas-pyconfig Events:       Much better. Then the final test; checking to see if batch submission works.   The program we want to run looks like:       proc python; submit; print('hello world') endsubmit; run;       The command that runs it looks like: sas-viya batch jobs submit-pgm --pgm-path /shared/gelcontent/gelcorp/shared/code/my_code.sas --context default --watch-output --wait-log-list --results-dir ~/   And that results in:       >>> The file set "JOB_20230406_102344_048_1" was created. >>> Uploading "test.sas". >>> The job "1dda5231-1978-4ab0-9da4-d0e264fdecde" was submitted. >>> Waiting for the job to complete... >>> Job started. 1 The SAS System Thursday, April 6, 2023 10:23:00 AM NOTE: Copyright (c) 2016 by SAS Institute Inc., Cary, NC, USA. NOTE: SAS (r) Proprietary Software V.04.00 (TS M0 MBCS3170) Licensed to THIS ORDER IS FOR SAS INTERNAL USE ONLY (SIMPLE), Site 00000000. NOTE: This session is executing on the Linux 3.10.0-1062.12.1.el7.x86_64 (LIN X64) platform. NOTE: Additional host information: Linux LIN X64 3.10.0-1062.12.1.el7.x86_64 #1 SMP Tue Feb 4 23:02:59 UTC 2020 x86_64 Red Hat Enterprise Linux release 8.7 (Ootpa) NOTE: SAS initialization used: real time 0.02 seconds cpu time 0.02 seconds NOTE: AUTOEXEC processing beginning; file is /opt/sas/viya/config/etc/sasgrid/default/autoexec.sas. NOTE: DATA statement used (Total process time): real time 0.00 seconds cpu time 0.00 seconds NOTE: AUTOEXEC processing completed. NOTE: The LOCKDOWN option has been set. SAS is now in the lockdown state. NOTE: The LOCKDOWN option has been set. SAS is now in the lockdown state. 1 proc python; 2 submit NOTE: Python initialized. Python 3.8.13 (default, Apr 4 2023, 04:53:36) [GCC 8.5.0 20210514 (Red Hat 8.5.0-16)] on linux Type "help", "copyright", "credits" or "license" for more information. >>> >>> 2 ! ; 3 print('hello world') 4 endsubmit; 5 run; 6 %let SAS_workpath = %sysfunc(pathname(work)); >>> hello world >>> NOTE: PROCEDURE PYTHON used (Total process time): real time 1.36 seconds cpu time 0.01 seconds NOTE: SAS Institute Inc., SAS Campus Drive, Cary, NC USA 27513-2414 NOTE: The SAS System used: real time 1.40 seconds cpu time 0.05 seconds 2 The SAS System Thursday, April 6, 2023 10:23:00 AM <<< Job ended. <<< Downloading the files in the file set to the following path: "/home/cloud-user//JOB_20230406_102344_048_1". <<< Downloading "test.log". <<< Downloading "test.sas". <<< Copying the log and list files from "/home/cloud-user//JOB_20230406_102344_048_1" to the current directory. <<< Copying "test.log". <<< Deleting the results directory. <<< Deleting the batch job. <<< Deleting the file set.       Success!   There's probably a solid enough case for having the open source configuration applied to both the compute and batch PodTemplates out-of-the-box with the python-transformer.yaml; which will obviously simplify and reduce the setup tasks required. I will update this post with the revised steps if the default setup changes.  Update: The open source configuration is available to SAS compute and batch sessions as of 2024.07, along with new, simplified documentation consolidating the instructions for configuring Python integration.   Thanks for reading. My thanks also to my colleaque @Bogdan_Teleuca for his contributions.   Find more articles from SAS Global Enablement and Learning here. ... View more

Effective alerting requires that the right people are notified of issues at the right time and via the most appropriate channels. SAS Viya Administrators and DevOps engineers using Alertmanager to manage alerts in the SAS Viya platform can integrate Alertmanager with several third party party applications to distribute alert notifications. While native support is built-in for some applications, many others are capable of receiving information via webhooks (e.g. alert notifications sent by Alertmanager). In this post, we'll look at how Alertmanager can integrate with Microsoft Teams via webhook, allowing alert notifications to be automatically posted to a Microsoft Teams channel when alerts are triggered.   There are several ways to set up the integration. In any case, as there is no native support for Teams integration in Alertmanager, and because Teams requires messages to be in a certain format (one that Alertmanager doesn't produce its alerts in), a third-party tool must be deployed to act as an intermediary. There are many to choose from, but in this case I used an open-source utility called prometheus-msteams. It is essentially a lightweight web server which receives alert messages from Alertmanager and sends them to a Teams channel in the required format.   But first, the Teams channel must be configured. Generate the unique incoming webhook URL (which is where prometheus-msteams will send messages) in the desired Teams channel by right-clicking on the channel name and clicking Connectors.     Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.    If it's not already added, click Add on the Incoming Webhook Connector. When it's added, click Configure.        Provide a name and then click create to generate the unique URL. Copy/take note of this, as it is required later.         Next, set up prometheus-msteams. There are several ways to deploy it, including as a binary or as a docker container. In my lab, I deployed it to my Kubernetes cluster (in the monitoring namespace deployed by SAS Viya Monitoring for Kubernetes) using the Helm chart. First, I added the chart repo as per the instructions with:   helm repo add prometheus-msteams https://prometheus-msteams.github.io/prometheus-msteams/   Then prepare the configuration file before deployment. This is where webhook URL from Teams must be specified.   tee ~/config.yaml > /dev/null << EOF replicaCount: 1 image: repository: quay.io/prometheusmsteams/prometheus-msteams tag: v1.5.1 connectors: # in alertmanager, this will be used as http://prometheus-msteams:2000/alertmanager – alertmanager: https://sasoffice365.webhook.office.com/webhookb2/e501090a-af3d-426c-bf35-2374d5e7d211@b1c14d5c-3625-45b3-a430-9552373a0c2f/IncomingWebhook/4b5a85a99819453aae87cbce67ea6469/07d2cf3f-f87b-4c4c-a3d6-2f10f2993322 # # Enable metrics for prometheus operator metrics: serviceMonitor: enabled: true additionalLabels: release: prometheus # change this accordingly scrapeInterval: 30s EOF   To deploy, run:   helm upgrade --install prometheus-msteams --namespace v4mmon -f config.yaml prometheus-msteams/prometheus-msteams   That results in:    Release "prometheus-msteams" does not exist. Installing it now. NAME: prometheus-msteams LAST DEPLOYED: Wed Mar 15 06:52:49 2023 NAMESPACE: v4mmon STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: ** Please be patient while the chart is being deployed ** To monitor the deployment, execute the following command: kubectl get pods -l app=prometheus-msteams --namespace v4mmon -w   Check that it has been deployed and is running:   kubectl get po -n v4mmon|grep msteams   The result:   prometheus-msteams-588777df8f-btmq2 1/1 Running 0 92s   The next step is to define a new receiver in the Alertmanager configuration for prometheus-msteams. In my lab, I made it the default receiver for all routes, so all alert notifications will get sent to Teams.   tee ~/alertmanager.yaml > /dev/null << EOF global: smtp_smarthost: $(hostname -f):1025 smtp_from: 'alertmanager@gelcorp.com' smtp_require_tls: false resolve_timeout: 5m route: receiver: prometheus-msteams group_wait: 30s group_interval: 5m repeat_interval: 12h receivers: - name: prometheus-msteams webhook_configs: - url: "http://10.96.3.151:1975/alertmanager" send_resolved: true EOF   Note that in this example, the URL is referring to the IP & port of the node where the prometheus-msteams pod is running. In my lab, I set up the prometheus-msteams service with a NodePort so that I could simulate test alerts using cURL (from outside the cluster).   Apply the updated configuration:   cat <<EOF | kubectl apply -f - apiVersion: v1 data: alertmanager.yaml: $(cat ~/alertmanager.yaml | base64 -w0) kind: Secret metadata: name: alertmanager-v4m-alertmanager namespace: v4mmon type: Opaque EOF   Once the Alertmanager config is applied, it may take a minute to take effect (hint: keep an eye on the Status page in the Alertmanager UI). When it does, notifications will start appearing as posts in the Teams channel when the alerts begin firing.        If you want to customize the way messages are posted, you can do that too by following the documented instructions and examples.   For more information refer to the GitHub page. Alternative approaches are also widely documented online.   If you'd rather use Slack than Teams, check out this previous post. Thanks for reading.   Find more articles from SAS Global Enablement and Learning here. ... View more