When deploying SAS Viya, the default behavior schedules both stateful and stateless workloads across any available node pools. Generally, this setup is ideal, as Kubernetes is tasked with efficiently allocating pods to the appropriate nodes. However, there may be some edge cases where stricter scheduling of workloads is required. For example, it may be desirable to segregate stateful and stateless workloads to better utilise resources across different node pools, such as in situations where solutions like SAS Visual Investigator (with pods that demand more storage or memory) are deployed across node pools of varying sizes. In this post, we'll talk about confining stateful workloads to stateful node pools, and stateless workloads exclusively to stateless node pools.   Ensuring Workload Separation   The primary challenge lies in controlling where different types of pods are scheduled. By default, both StatefulSet (stateful) and Deployment (stateless) pods tolerate both stateful and stateless node classes. This flexibility can sometimes result in stateful pods being scheduled on stateless nodes, which might not have appropriate resource allocations, and could potentially lead to throttling or pod eviction during high activity from stateless pods.   Two approaches can be taken to address this challenge.   Setting up required node affinity   Node affinity allows us to define rules that specify where pods should be scheduled, based on the labels applied to nodes. This approach offers more granularity and flexibility, allowing for strict scheduling of workloads on particular nodes. In this case, we can switch the affinity rules from 'preferred' to 'required', as covered by my colleague Raphaël Poumarede in his post.   Begin by labeling the nodes to distinguish between stateful and stateless node pools.   kubectl label nodes workload.sas.com/class=stateful kubectl label nodes workload.sas.com/class=stateless   Add the following node affinity rules to StatefulSets to ensure these pods are scheduled only on stateful nodes.   spec: template: spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: workload.sas.com/class operator: In values: - stateful   Similarly, ensure that stateless pods (Deployments) are scheduled only on stateless nodes by modifying the YAML templates as follows:   spec: template: spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: workload.sas.com/class operator: In values: - stateless   Rebuild and redeploy to apply these changes.   Modifying tolerations   Tolerations work with taints applied to nodes, determining which pods are allowed to run on nodes with specific taints. By default, stateful and stateless pods tolerate both node classes. Modifying tolerations provides an alternative, simpler method of enforcing workload placement.   We can modify (remove) the tolerations in the YAML files that define the pods to achieve segregation of stateless and stateful workloads.   Here’s a sample patch to remove the stateful toleration from all Deployment resources to confine stateless pods to nodes optimised for stateless workloads.   --- apiVersion: builtin kind: PatchTransformer metadata: name: remove-stateful-toleration patch: |- - op: remove path: /spec/template/spec/tolerations value: - effect: NoSchedule key: workload.sas.com/class operator: Equal value: stateful target: group: apps kind: Deployment version: v1   An update to kustomization.yaml is also required:   ... transformers: - remove-stateful-toleration.yaml # Path to your patch file   Once applied, this patch will ensure that deployment resources are no longer scheduled on stateful nodes, improving the separation of workloads and enforce the changes across the environment.   Choosing the right approach   Both approaches outlined here offer effective methods for ensuring that stateful and stateless workloads are scheduled on the appropriate node pools in SAS Viya deployments. Other types workloads can also be segregated as desired. Note, however, that the default behaviour of scheduling across any available node pool is usually perfectly appropriate.   Required affinity provides more flexibility and control to enforce scheduling rules based on node labels, which can be easily adjusted if new node pools are added or removed. There is, however, an added administration overhead of maintaining node labels.   Modifying tolerations offers a simpler method for restricting pod placement by removing tolerations for nodes to repel unsuitable pods, but there is somewhat less flexibility, such as in handling changes to node pools.   Either method (or a combination of both methods) can be used to ensure stricter workload distribution in your SAS Viya environment with pods more efficiently distributed across node pools.   My thanks to Raphaël Poumarede, Harshit Soni and Henrique Lima for their contributions.       Find more articles from SAS Global Enablement and Learning here. ... View more

Auditing capabilities available in SAS Viya are essential for understanding user behavior and ensuring that resources are used appropriately. In this post, we'll demonstrate a method of tracking and viewing user access to SAS Visual Analytics reports using audit data in SAS Viya.   In SAS Viya, the Audit service is responsible for the capture and storage of records relating to user activity. By default, the Audit service does not capture successful 'read' events, so to capture information about who accessed a report, a simple change is first required to the sas.audit.record configuration instance in SAS Environment Manager's Configuration area.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   The audit.recording.level property can be adjusted to specify which events to capture. To enable the capture of successful 'reads', change the audit.record.level to a setting of High.     Note, though, that this can significantly increase the volume of data that is captured and stored. Your mileage may vary, but it’s important to balance capturing detailed information with managing storage requirements. One option is to define the service.list property, where you can specify a comma-separated list of application/service names for which successful 'read' events will generate audit records. It might also be prudent to adjust the retention periods for audit records.   Once that's done, the next time a user accesses a report, an audit record is created. In fact, many audit records are created. Suppose a user has viewed the sample Warranty Analysis report. The sas-viya --output text audit list --details command will return something like:   ID Time Stamp Type Action State Description User ID Application Remote Address Administrative Action URI 312c1b34-59b8-4fd9-97c8-016f9cb2ded2 2024-05-23T06:17:02.788Z resource read success Ahmed reports 127.0.0.1 true /Products/SAS Visual Analytics/Samples/Retail Insights.report 31d36d6b-6867-4840-b044-80b09a76c043 2024-05-23T06:17:02.69Z resource read success geladm reports 127.0.0.1 true /Products/SAS Visual Analytics/Samples/Warranty Analysis.report ceb2fadb-0bf2-4720-8e41-1eebb69a42c1 2024-05-23T06:17:02.596Z resource read success geladm reports 127.0.0.1 true /Products/SAS Visual Analytics/Samples/Water Consumption and Monitoring.report 4174fd1e-e0c9-498d-9394-a7ba955d58b1 2024-05-23T06:17:02.512Z resource read success geladm reports 127.0.0.1 true /Products/SAS Visual Analytics/Samples/Warranty Analysis.report 2841747c-fcdc-42b3-97cb-38bbcacd2bbd 2024-05-23T06:17:02.415Z resource read success geladm reports 127.0.0.1 true /Products/SAS Visual Analytics/Samples/Retail Insights.report c8e2e71c-dc49-4acf-bef4-42f57f27d1d1 2024-05-23T06:17:02.33Z resource read success geladm reports 127.0.0.1 true /Products/SAS Visual Analytics/Samples/Warranty Analysis.report bd9a5747-3871-470c-8010-3cbfe4c87ef1 2024-05-23T06:17:02.22Z resource read success geladm reports 127.0.0.1 true /Products/SAS Visual Analytics/Samples/Warranty Analysis.report 0540029e-08fe-4e59-9383-14cd2e7e40e7 2024-05-23T06:17:02.104Z resource read success Ahmed reports 127.0.0.1 true /Products/SAS Visual Analytics/Samples/Warranty Analysis.report 1417b61c-657a-4015-87d8-1b77a15bcb69 2024-05-23T06:16:42.226Z resource read success Ahmed reports 10.42.1.39 false /Products/SAS Visual Analytics/Samples/Warranty Analysis.report   Note that not only does Warranty Analysis appear multiple times as a successful read event, but other reports that we didn't open are also listed. This is because a "read" action is more than just the act of opening a report in SAS Visual Analytics. For example, when report thumbnails are rendered, these are also registered as "read" actions for the user, even if that user never actually opened the reports at that time.   How then do we differentiate between "reads" when a user physically opens a report and these other "reads"? To do so, we can use the RemoteAddress field (which is shown when the --details flag is added to the CLI command). The report that uniquely identifies the actual user action of opening the report contains the the podIP of the sas-folders pod as the value for the RemoteAddress field. We can get that with the command: kubectl get pod -l app=sas-folders -o custom-columns=NAME:metadata.name,IP:status.podIP   In my lab environment, that matches the last record displayed in the output above. The CLI command used to query the audit data could be modified to include a filter such that RemoteAddress equals the IP address of your sas-folders pod. Note that this behaviour is slightly different to the behaviour in SAS Viya 3.x.   Auditing the usage and access of SAS Visual Analytics reports is a critical aspect of managing and securing your SAS Viya environment. By leveraging the detailed audit data available, you can effectively monitor report access and usage and swiftly address any issues that arise. Implementing robust auditing practices not only aids in compliance but also promotes accountability and transparency within your organization.   For more information on auditing in SAS Viya, refer to the official SAS documentation.     Find more articles from SAS Global Enablement and Learning here. ... View more

In the world of monitoring and observability, the ability to detect and respond to issues promptly is paramount. Alertmanager and Grafana, both deployed with SAS Viya Monitoring for Kubernetes, possess alerting capabilities. While both serve the purpose of alerting, they have distinct features and approaches that cater to different needs. In this blog post, we'll endeavour to compare Alertmanager and Grafana by their alerting functions and provide some points to consider for when you might want to use one over the other.   Alertmanager's key features   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   Alertmanager is an open-source alerting tool that works seamlessly with Prometheus for a comprehensive metric-based monitoring and alerting toolkit. It is designed to handle alerts sent by client applications such as Prometheus and then manage these alerts by grouping, deduplicating, and routing them to various integrations like email, PagerDuty, Slack, and Microsoft Teams. Both are deployed together with SAS Viya Monitoring for Kubernetes. Since Alertmanager is tightly integrated with Prometheus, it leverages PromQL, Prometheus' powerful query language for defining alerting rules. This makes it easy to create sophisticated alerting conditions based on Prometheus metrics.   Alerts are created as PrometheusRules, and evaluated against metric data stored in Prometheus. Rules use YAML; easy to use, edit, store and deploy. Alertmanager allows users to define complex alert routing and inhibition rules based on labels attached to alerts. This flexibility enables fine-grained control over which alerts are sent to which receivers. For more details on how the routing process works, check out thisearlier post. Alertmanager provides a silence mechanism that allows users to temporarily mute specific alerts or entire groups of alerts.; useful when dealing with issues and addressing underlying causes. Alerts can also be inhibited in advance during planned outages, such as during maintenance activities. Alertmanager can be deployed in a highly available configuration to ensure reliability. It supports clustering and replication for fault tolerance.   Grafana's key features     Grafana is a popular open-source platform for monitoring and observability that offers a wide range of features, including visualization, dashboarding, and alerting. While Grafana is primarily known for its visualization capabilities, it also provides robust alerting functions. Older versions of Grafana required alerts to be created on dashboards, which was quite restrictive. While this is still possible, the Alerting facility that is now included offers much more comprehensive and flexible capabilities.   Grafana allows users to define alerting rules visually using its intuitive web interface. Users can set thresholds, conditions, and notification channels for triggering alerts directly in the UI (no need to write YAML or run kubectl commands). Grafana's alerting feature seamlessly integrates with its visualization capabilities. Users can create alerts directly from dashboards and visualize alerting states alongside other metrics. Grafana supports a variety of data sources; not just Prometheus. This flexibility enables users to create alerts based on data from different monitoring systems. Useful if other existing tools are already in use. Grafana keeps a history of triggered alerts, making it easy to track the lifecycle of alerts over time. Additionally, users can annotate graphs with alert events for better context.   Comparison   Authentication   Grafana requires users to log in with a username and password. Alertmanager has no such authentication built in; if the URL is accessible, anyone can use it. There are other ways to 'control' access to Alertmanager, but they need careful planning and consideration. Here is one example. Certainly, none are as convenient as Grafana's simple login prompt.   Ease of Use   Grafana's web interface provides an intuitive way to create and manage alerts (visually), making it more accessible to users who are not familiar with Prometheus' query language. On the other hand, Alertmanager's configuration may require more expertise due to its reliance on Prometheus' query language.     Default alerts   Prometheus includes a number of pre-defined alerts for Kubernetes applications when SAS Viya Monitoring for Kubernetes is deployed. Very useful for alerting on generally common issues, and no additional setup is required beyond defining notification channels for them. Grafana conveniently also displays these alerts, but interacting with them (e.g. silencing them when they fire) is a little less intuitive for these than it is for Grafana-managed alerts.     Integration   Both AlertManager and Grafana offer integration with various notification channels such as email, Slack, and PagerDuty. However, Grafana's integration capabilities extend beyond alerting to include visualization and dashboarding (Grafana is first a monitoring tool after all). There are also options for integrating the two together; for instance, to send Grafana-managed alerts to Alertmanager when they begin firing.   Scalability   Alertmanager can easily be scaled horizontally for high availability to handle large volumes of alerts; in fact, it's designed to be clustered. Grafana's scalability depends on the underlying data source it's connected to, but it can also scaled with a bit of effort. With SAS Viya Monitoring for Kubernetes, though, both are deployed as standalone instances by default.   Customisation   Alertmanager provides more fine-grained control over alert routing and inhibition rules, making it suitable for complex alerting workflows. It also offers options to customise nearly every aspect of the alert notifications. Grafana, on the other hand, offers greater flexibility in terms of visualization and dashboard customization, which Alertmanager lacks.   Conclusion   Both AlertManager and Grafana are powerful tools for alerting in the context of monitoring and observability. The choice between the two (or both!) depends on factors such as familiarity with Prometheus, integration and security requirements, and needs for advanced alerting features. Grafana may generally be considered more convenient for administrators with basic alerting requirements or whom are new to observability tools, whereas the expertise required to best leverage Alertmanager's powerful capabilities may be better suited to organisations that already use and have experience with it and related tools (e.g. Prometheus). Ultimately, organisations should evaluate their specific use cases and requirements to determine which tool best fits their needs. ... View more

The event-driven architecture in SAS Viya 3.5 (and 3.4) offers the capability for effective auditing of system resources, report activity, data access, and more. In this post, we’ll revisit the options available for auditing user access and highlight important caveats and considerations. This revised version of an earlier article has been updated to account for some changes in the underlying processes to provide a clearer picture.   CAUTION: Important considerations   Before adopting the approach to auditing user sessions documented in this post, beware these important cautions about using the EV Datamart data captured by the SAS Operations Infrastructure for this purpose: The data is not originally intended as 'audit' and it does not contain data integrity and security features that true audit data should. The data is derived from different data sources in which information is recorded differently, and matching user sessions with this data can be imprecise. It is not suitable for real-time or near-real-time purposes due to (potentially significant) latency in data being written to disk. Matching session activity like logons and logoffs using the session ID may not always work as expected. While the session ID is generally unique, it is not guaranteed to be unique. There are a small number of situations and edge-cases where a given session ID might be reused. SAS Viya 3.5 versions released in December 2023 (and later) address an issue where the session signature was not recorded for a login event, resulting in errors with the AUTHENTICATIONS table data. The information outlined in this post is a potential workaround for simplifying the process for auditing user session, but the caveats listed above must be noted as they present some potential challenges with this approach. Proceed with caution.   Information about logins and user sessions is captured by the Audit service in the AUDIT.security_audit tables in the SAS Infrastructure Data Server. The default behaviour, as defined in the security type properties in the sas.audit.record configuration instance, is to capture read failures only on security actions.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.    When trying to audit user login attempts, the login event provides information about (successful and failed) login attempts coming from the SASLogon application.   Let’s look at an example. Below, we use the sas-admin CLI’s audit plugin to see that user Ahmed logged in at 07:31 AM on the 25th of October (note that the timestamp is in UTC in PostgreSQL, but is converted to local time when a SAS Operations Infrastructure ETL job loads it to the AUDIT table in CAS).   /opt/sas/viya/home/bin/sas-admin --output text audit list --user-id Ahmed --sort-by ~timeStamp   ID Time Stamp Action State User ID Application URI c72c4533-886d-493b-9ef2-1dcff4ff4674 2019-10-25T07:57:54.776Z SessionDestroyed success Ahmed SASLogon bb748c54-6cb3-4edf-adbe-fce516c5786c 2019-10-25T07:31:21.014Z update success Ahmed reportImages /reportImages/jobs/4d7a64d8-1839-4d5b-a562-b140cdfc1f22/state caf1264d-4d37-444b-af45-4e339edc5dc6 2019-10-25T07:31:20.999Z create success Ahmed reportRenderer /reportRenderer/images 713e18ac-34db-4115-92aa-35f7662b0909 2019-10-25T07:31:20.649Z read success Ahmed reportData /reportData/commons/settings c5df6523-eb4f-42fe-b89b-fbdcc73d1c56 2019-10-25T07:31:20.298Z create success Ahmed reportPackages /reportPackages/jobs/b7325111-3f5a-4659-84bf-d1e59bdfbc74 e8578d9f-73b1-4dea-a94e-048a9eb9a72d 2019-10-25T07:31:20.030Z create success Ahmed reportData /reportData/jobs/61c55ce8-2b0a-4989-a14f-e1a5395b3b86 5aec043f-c2f8-46b6-a758-bc040abf1f69 2019-10-25T07:31:17.990Z create success Ahmed reportData /reportData/metadataSupplements fc7c64d8-bf75-40f5-a851-4afa5dc29beb 2019-10-25T07:31:16.141Z create success Ahmed reportImages /reportImages/jobs/4d7a64d8-1839-4d5b-a562-b140cdfc1f22 f9ad69f9-22b8-447b-82c2-45ebeb462409 2019-10-25T07:31:14.523Z read success Ahmed reportData /reportData/commons/settings 5dc96422-833c-496c-bd65-64659faf76d0 2019-10-25T07:31:08.641Z read failure Ahmed preferences /preferences/preferences/@currentUser/FormatLocale.DefaultLocale 5da5acf5-4d6b-47b3-9da5-c30ea7c3d946 2019-10-25T07:31:08.096Z read failure Ahmed preferences /preferences/preferences/@currentUser/_hc_xx_ 4a4c470f-afe9-469c-af3a-3fdb6d5f951e 2019-10-25T07:31:07.082Z read failure Ahmed preferences /preferences/preferences/@currentUser/FormatLocale.DefaultLocale b66f6eed-8ae8-47a3-8d3a-60e23df5cec9 2019-10-25T07:31:05.463Z login success Ahmed SASLogon   When a user logs off gracefully, a SessionDestroyed action is registered indicating the termination of the session as shown above.   If a user closes the browser instead of logging off, the SessionDestroyed does not get created in the audit records until the session times out. Session timeout duration is configurable.Set Time-out Interval for SAS Viya Web Applications in SAS® Viya® 3.5 Administration: Configuration ....   Some applications generate their own audit data when a user logs in. For example, SASStudio V creates a Compute server session when a user logs in, and an audit record similar to the following is captured.     /opt/sas/viya/home/bin/sas-admin audit list --user-id Ahmed --sort-by ~timeStamp --details --application compsrv   ID Time Stamp Type Action State Description User ID Application Remote Address URI 1967127d-87da-4656-922f-ed3b3bf1733d 2019-10-23T11:47:42.122Z resource start success 92ce6360-b23c-485b-94dc-48e5bd681bdd:https://intviya02.race.sas.com:38934/compute/servers/92ce6360-b23c-485b-94dc-48e5bd681bdd Ahmed compsrv   When the compute session is terminated, an audit record is captured indicating the deletion of the session (with the Description field indicating to the ID of the session).   /opt/sas/viya/home/bin/sas-admin audit list --user-id Ahmed --sort-by ~timeStamp --details --application compute   ID Time Stamp Type Action State Description User ID Application Remote Address URI 5ebd10ca-b4e1-4e7e-b71d-2874bfa672ca 2019-10-23T07:50:36.257Z resource delete success Deleted compute server 92ce6360-b23c-485b-94dc-48e5bd681bdd Ahmed compute /compute/servers/92ce6360-b23c-485b-94dc-48e5bd681bdd c9a1fc4a-48ea-4afa-a456-45e890571e42 2019-10-23T07:50:36.132Z resource delete success Deleted compute server session 92ce6360-b23c-485b-94dc-48e5bd681bdd-ses0000 Ahmed compute /compute/sessions/92ce6360-b23c-485b-94dc-48e5bd681bdd-ses0000   User sessions are assigned a session signature when they are established. The session signature (session_sig) property links the login event with the corresponding SessionDestroyed event. However, there are some important considerations.   First, refer to the caveats listed at the beginning of this post about potential issues with relying on the session signature for this purpose.   Second, the session signature is not visible when surfacing audit records using the CLI. This information is recorded in TSV files in the EV Datamart directory (/opt/sas/viya/config/var/lib/evmsvrops/evdm/application/security) on the Viya server. When a session is started:   2 34dca2cc-fcc9-4b6d-8c72-32d2958c9320 security application/vnd.sas.event.security 2019-10-15T06:21:18.973000+00:00 sasadm sas-deployment-id:dml5YQ==,sas-event-source:U0FTTG9nb24=,session_sig:NTNlZmNlZGE= security action:bG9naW4=,actionState:U1VDQ0VTUw==   Note the session_sig value. When the session ends, another record will be generated in the same file, referring to the original signature (orig_session_sig) for that session.   2 b41e8882-192d-4657-8fee-4cb04a96abda security application/vnd.sas.event.security 2019-10-15T07:02:30.282000+00:00 sasadm orig_session_sig:NTNlZmNlZGE=,sas-deployment-id:dml5YQ==,sas-event-source:U0FTTG9nb24=,session_sig:NDk0MGZmNw== security action:U2Vzc2lvbkRlc3Ryb3llZA==,actionState:U1VDQ0VTUw==   Fortunately, the legwork for extracting this information has already been done. Session information is extrapolated from these files and stored in the authentications subject table (opt/sas/viya/config/var/lib/evmsvrops/evdm/subjects/authentications.sas7bdat). Information is captured in this table for metering purposes, a secondary effect is that it provides a really nice way (with the caveats listed above!) to audit user session information. In the example below, several user sessions are listed, each with a datetime stamp of the login action, the session signature, and the duration of the session (in seconds).   Note that user Sasha’s session has a duration of 0.00 seconds. This points to an active user session (but there may be exceptions - listed above -  due to latency), indicating that Sasha has not yet logged out (SessionDestroyed action has not occurred for the session with this signature).   A job runs every 5 minutes to update this table, and another job runs daily to clear records older than 10 days. Update: To change the number of days (e.g. from 10 to 14 days) for which to capture records, execute:   /opt/sas/viya/home/bin/ops-dm-admin set EMI_DELETE_TSVZIP_DAYS=14   We can query this table to get some more insights about user sessions. Can you use it to figure out who was logged in at a certain time, for example?   A basic SAS program can be submitted to help answer that question.   /* Specify query time below in datetime. format */ %let qrytime='15OCT19:08:39:18'dt; libname evdm '/opt/sas/viya/config/var/lib/evmsvrops/evdm/subjects' access=readonly; data activesessions; set evdm.authentications; endtime=datetime+duration; format datetime datetime.; format endtime datetime.; where &qrytime between datetime and &endtime or (&qrytime >= datetime and (duration=0 and session_sig ne ''); run; proc print data=activesessions; run;   Note that this code sample is for demonstration purposes only, and there may be more efficient or more accurate methods to extract the required user session information.   Thanks to my colleagues Greg Smith, Carl Sommer, Fred Forst and Mike Roda for their contributions to this and the original version of this article, and thank you for reading! ... View more

PostgreSQL 15 is available in the November 2023 release of SAS Viya 3.5. SAS strongly recommends that customers upgrade their deployed PostgreSQL instance to version 15. Customers running SAS Viya 3.5 should check to see if they already have version 15 deployed, and if not, should consider upgrading in order to stay comfortably within PostgreSQL's support policy and to stay current with security fixes and patches. In this post, we will demonstrate how to check what version you are running in your environment, and highlight the steps you must perform to upgrade your deployed version of PostgreSQL to version 15.   If you are running a version of SAS Viya older than Viya 3.5, you must first upgrade your environment to SAS Viya 3.5 if you want to obtain PostgreSQL 15. Specifically, you should upgrade to at least Ship Event 23w44, which is the earliest release of SAS Viya 3.5 that contains PostgreSQL 15. This means that to get PostgreSQL 15, you must obtain a new order from SAS and then perform an upgrade. Any new deployments of SAS Viya 3.5 from 23w44 or later will deploy PostgreSQL version 15 by default. Similarly, if you are running a version of Viya 3.5 earlier than Ship Event 23w44, you need to upgrade to at least that release to get PostgreSQL 15. The simplest way to verify the version you are running is to directly query the version of PostgreSQL deployed in your environment by doing the following:   Determine the deployment target(s) for PostgreSQL in your environment In you inventory.ini file, look in the [sasdatasvrc] and [pgpoolc] (if applicable) host groups to identify the target machines, and then look in the [host_definitions] at the beginning of the file to view host information for those machines. Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   Check if the PostgreSQL 15 binaries have been deployed If SAS Viya is deployed on Linux, run the following command on the target machines: ls /opt/sas/viya/home/postgresql15 If not found, you do not yet have PostgreSQL 15 deployed.   Or if deployed on Windows, run the following in Powershell: dir "C:\Program Files\SAS\Viya\lib\postgresql15\bin"   If not found, you have not yet deployed PostgreSQL 15.   In these cases, you must contact SAS to obtain a new SAS Viya 3.5 software order from Ship Event 23w44 or later to upgrade to PostgreSQL 15. Once you have the new order, upgrade your SAS Viya deployment by following the documented instructions, remembering to perform the all-important pre-upgrade and post-upgrade steps for PostgreSQL. If you do have the PostgreSQL 15 binaries deployed, continue to the next step.   Verify whether your SAS Viya deployment is actually running the deployed PostgreSQL 15.   It is possible that your environment has PostgreSQL 15 deployed, but post-upgrade steps to configure it haven’t been performed. As an additional check, verify whether your SAS Viya deployment is actually running the deployed PostgreSQL 15.   On Linux, run the following as sudo on one of the target machines identified earlier: sudo cat /opt/sas/viya/config/data/sasdatasvrc/postgres/node0/PG_VERSION On Windows, run the following in Powershell:   cat C:\ProgramData\SAS\Viya\data\sasdatasvrc\postgres\node0\PG_VERSION A value of 15 listed in the PG_VERSION file indicates that PostgreSQL 15 is running and in use by SAS Viya. You can stop reading here, because all is as it should be!   However, if the listed version is not 15, some further action is required. We need to check to see if the latest PostgreSQL and related RPMs/MSIs are installed. On Linux, run the following to check the deployed RPM versions for PostgreSQL components:    cd /etc/init.d rpm -qa | egrep 'sasdata|sas-postgres|pool'   The versions displayed might resemble the below (the latter part of the values displayed for each RPM is irrelevant and has been omitted):   sas-sasdatasvrc-0.8.1-XXXXXXXXX.XXXXXXXX sas-postgresql-secure-libs-15-15.4-XXXXXXXXX.XXXXXXXX sas-pgpoolc-0.7.0-XXXXXXXXX.XXXXXXXX sas-postgresql-secure-15-15.4-XXXXXXXXX.XXXXXXXX sas-pgpool-II-44-4.4.4-XXXXXXXXX.XXXXXXXX If the versions shown above are displayed, this indicates you have deployed PostgreSQL 15 but not configured it yet. Some additional steps are required to reconfigure PostgreSQL and pgpool and complete the upgrade to version 15. Review the post-upgrade steps and completed all documented tasks.   If you do not have those versions, contact SAS to obtain a new SAS Viya software order from Ship Event 23w44 and upgrade your SAS Viya deployment in order to obtain PostgreSQL 15 Similarly, on Windows deployments, run the following in Powershell to check the MSI versions:    dir /powershell-deployment\Downloads   If sas-datasvrc15-15.4.0.0.msi   is listed, you have PostgreSQL15 deployed but not configured yet. Again, review the post-upgrade steps and completed all documented tasks. If you do not have that version, contact SAS to obtain a new SAS Viya software order from Ship Event 23w44 and upgrade your SAS Viya deployment in order to obtain PostgreSQL 15. A couple of additional points to consider:   Perform a full backup after updating your SAS Viya deployment After upgrading PostgreSQL, do not restore backups that were taken in older versions of PostgreSQL. If you have upgraded to PostgreSQL version 15, and you wish to then migrate to SAS Viya 4, you must move to a supported target version of SAS Viya 4 using an external instance of PostgreSQL 15.  Thank you for reading. My thanks also to my colleagues Cindy Taylor, Joseph Cho and Fred Perry for their contributions.        ... View more

Staying up-to-date with SAS Viya software is more important than ever. Updating regularly will give you the latest features and fixes, but also ensures that your deployment doesn't fall out of Standard Support. Before you set sail on your voyage to update to new SAS Viya version, it's essential to prepare properly to ensure a safe and smooth transition. In this article, we'll explore the important pre-upgrade tasks that will help reduce the risk of unforeseen issues and ensure you are set up for a successful update.   Pre-update Checklist   The SAS Viya Platform Administration documentation lists the SAS Viya platform versions that are in Standard Support and the recommended update paths to the latest version. In fact, the task of checking the support status of your deployment is the very first item in the Pre-update checklist of essential tasks that SAS now provides, which should be consulted before each update. If skipping versions or switching cadence as part of your update, take care to correctly identify interim versions which may be relevant; you will need to consult the Deployment Notes and execute the documented Before Deployment Commands for these.   In addition to checking your SAS Viya software support status, the checklist includes a task to verify your deployed Kubernetes version. Carefully read the documentation to see if you also need to upgrade to a supported version of Kubernetes or Kustomize.    Additionally, if your deployment includes it, consider updating the SAS Deployment Operator with each SAS Viya software update. Updates are backwards compatible, so running the latest version is a good practice. Note that if you have the SAS Deployment Operator deployed, you must use it to perform the SAS Viya update. If you used the sas-orchestration deploy method for the initial install, you must use that. It is not possible to mix and match deployment methods for update tasks.   A good practice is to run an inventory scan report before and after any major platform changes, including software updates. The two scans can be compared to verify that your content was not affected/changed during the update.   The inventory scan should also be run prior to a backup (and if required, after a restore). Pre-update backups should include an ad-hoc SAS Deployment Backup, backups of the $deploy directory and kustomization file, and if applicable, the SASDeployment CR file.   Next on the checklist is Deployment Assets. These should be retrieved for the target version you are moving to. Once downloaded, take a look at the README.md files for products you have deployed - this step is particularly important if you are adding new software as part of an upgrade, as there may be additional kustomizations or other manual steps that need to be performed for those new products. Note that assets automatically include the latest available license.   There are additional tasks in the checklist for multi-tenant environments and deployments that include SingleStore or multiple CAS servers. Be sure to work diligently through the list to ensure all relevant tasks are appropriately addressed.   Additional Tasks   Updates should be carefully co-ordinated with relevant technical resources to complete pre-requisite tasks. You may need to liaise and work with your Kubernetes administrator, security teams, change managers, end users, etc. to ensure you are ready for the update to begin. For example, you may need resources assigned to perform supplementary tasks ahead of time, such as:   providing sufficient access to enable you to perform the software update upgrading Kubernetes software (if required) verifying that the cluster has sufficient available resources to enable duplicates of each pod to run temporarily during the update notifying users of potential disruption updating the mirror registry, if applicable retrieving the output of the update checker job (bonus points for automating this)   Each organisation may have unique considerations, so seek further guidance from SAS Technical Support if necessary. With a well-executed pre-upgrade plan, you will avoid pitfalls and minimise disruptions on your way to the latest version of SAS Viya.     Find more articles from SAS Global Enablement and Learning here. ... View more

Grafana provides comprehensive insights into your SAS Viya deployment's health and performance using metrics collected by Prometheus. New capabilites were recently added to Grafana to improve its alerting capabilities. In this post, we will walk you through the process of configuring alerting for your SAS Viya deployment in Grafana.   Grafana requires Prometheus or another compatible data source set up to gather metrics from SAS Viya components. This is the default when SAS Viya Monitoring for Kubernetes is deployed.   The Grafana configuration is stored in a configMap (v4m-grafana) in the monitoring namespace. The config stored here includes things like your SMTP server host and port. If Grafana is already deployed, running a set of commands like the below will toggle on email connectivity and specify your mail server details in the configMap to allow Grafana to send alert notifications via email.   # get configmap in .yaml kubectl --namespace monitoring get configmap v4m-grafana --output=yaml > /tmp/v4m-grafana-configmap.yaml # update yaml file to enable smtp and enter mail server host:port sed -r -i "/kind: ConfigMap/ i\ \ \ \ [smtp] \\ \ \ \ enabled = true \\ \ \ \ \ host = mailserver.gelcorp.sas.com:25" /tmp/v4m-grafana-configmap.yaml # apply updated yaml kubectl -n monitoring replace --filename=/tmp/v4m-grafana-configmap.yaml # restart grafana kubectl -n ${_monitoringNS} delete po -l app.kubernetes.io/name=grafana   If Grafana isn't deployed, it's possible to specify these config parameters (and others) pre-deployment in the user.env file provided with SAS Viya Monitoring for Kubernetes.   Before actually getting to creating alerts, you must configure the notification channels to receive alerts. First, log on to Grafana and navigate to the "Alerting" section by clicking on the bell icon in the left sidebar.   Note that the main "Alert rules" page displays any other alerts configured in Prometheus. While there are limited options for interacting with these alerts, having the ability to view all alerts in one interface is very convenient. The best part is that when SAS Viya Monitoring for Kubernetes deploys Grafana, it is already configured to use Prometheus as a data source for metrics, so these alerts appear automatically without any additional config.   Click on the "Contact points" tab. This is where we configure our channels. A default one, grafana-default-email, is pre-defined for us. Click the pencil icon to open the edit page, where you can select the contact point type (Email) and then specify the destination address. Click the Test button after doing so to make sure everything is set up correctly for the recipient to receive the notification.     Save the changes.   Now head back to the Alert rules tab. The new alert we create in this page will be a Grafana managed alert (i.e. not a PrometheusRule). Click on "+ New alert rule" under Grafana alerts to begin adding the alert. Define the alert conditions based on SAS Viya metrics collected by Prometheus (using either the builder or code methods). Grafana's interface for creating alerts provides a rich set of options for controlling all aspects of your alert condition, and includes the best features of the Prometheus Expression Browser and then some. Specify or select the appropriate metrics, choose how they are filtered and evaluated, and then set the threshold values and evaluation frequency. For example, you could set a rule to trigger when memory usage exceeds a certain value for a specified duration.   Craft an informative rule name and message as well as other descriptive attributes that clearly indicate the issue. You will also be required to select (or create) a folder and group for your rule. In the "Notifications" section, select the key-value labels that will connect the alert to contact points (the equivalent of Alertmanager's routing function).   Save the alert. When the evaluation period lapses, and the alert condition is met, the alert will begin firing and a notification will be sent.         Adjust the threshold values, duration, or other settings as needed based on your alerting strategy and requirements.   We'll look at next steps and additional options in a future post. Stay tuned.   Thank you for reading.   ... View more

As part of ongoing efforts to enhance SAS Viya administration capabilities, we will introduce in this blog post a new program within the pyviyatools repository. getactivityrecords.py is a Python script designed to retrieve and analyze activity records from the SAS Viya environment. These activity records are a kind of audit record that provide insights into the operations and usage patterns of your SAS Viya deployment, enabling administrators and audit compliance folks to better understand at a high-level how the Viya deployment is being used by end users.   The concept of activities has been introduced within SAS Viya's audit service to provide a way to inject higher-level business logic into the auditing process. Traditionally, auditing focuses on low-level technical details, providing limited visibility into the actual usage patterns of different endpoints, making it somewhat challenging to derive meaningful insights. With the introduction of activities, the auditing framework gains a new, higher-level perspective to make auditing more intuitive and understandable. This improved understanding allows for better analysis, troubleshooting, and decision-making based on the audit data.   The audit service routinely loads activity data, which is stored in PostgreSQL, into the SystemData.AUDIT_ACTIVITIES table in CAS for reporting. The User Activity VA report provides a way for users to visualise audit data, but there is not yet a way to do the same for activity data, nor to view the raw activity records themselves.   Enter getactivityrecords.py, the latest addition to the growing pyviyatools repository.   To get started with getactivityrecords.py, you can simply clone the pyviyatools GitHub repository and install the necessary dependencies. When it is run, getactivityrecords.py leverages the SAS Viya REST API to retrieve activity records from the SAS Infrastructure Data server. It also allows you to specify the time range and filters to narrow down the scope of the records you want to retrieve. These records can not yet be surfaced from the sas-viya CLI's audit plug-in, which instead retrieves only the 'regular' audit records.   The program supports multiple output formats, including CSV (default), JSON, and Pandas DataFrames. This flexibility allows you to seamlessly integrate the retrieved activity records into your preferred reporting tools or visualization platforms. Leveraging scheduling tools like cron, you can automate the execution of getactivityrecords.py at regular intervals, enabling the continuous capture of activities within the SAS Viya environment.   The simplest way to execute the program is to simply run it without any additional parameters: getactivityrecords.py     Output:   id ,type ,action ,administrativeAction ,state ,user ,application ,timeStamp ,remoteAddress "2e88b242-1e6b-4aae-bab0-5cdb740d5aee","resource","create","False","success","dagentsrv-gelcorp","identities","2023-06-13T09:15:34.471Z","10.42.1.13" "8d6a74dc-5219-41b4-aaf7-538b24a8e41a","resource","create","False","success","dagentsrv-gelcorp","identities","2023-06-13T09:15:35.877Z","10.42.1.13" "e40bf1ec-47db-4d95-89a0-aa7d3a5204a4","security","login","False","success","geladm","SASLogon","2023-06-13T09:21:54.678Z","127.0.0.1" "efd0af8e-6439-4a3e-97e9-fe50efa5a163","security","login","False","success","geladm","SASLogon","2023-06-13T09:21:55.61Z","127.0.0.1" "5394eeb4-a321-4a3a-bd2c-0b7a4cb01438","security","login","False","success","geladm","SASLogon","2023-06-13T09:22:02.093Z","127.0.0.1" "37e04c46-693f-4899-bb74-3a84a593f184","security","login","False","success","geladm","SASLogon","2023-06-13T09:22:03.771Z","127.0.0.1" "0d4d1a45-2b12-4399-89ba-57de5a6753c7","security","login","False","success","geladm","SASLogon","2023-06-13T09:22:04.449Z","127.0.0.1" "f2ce150d-613b-48c4-812f-7cb86ecad433","security","login","False","success","geladm","SASLogon","2023-06-13T09:22:05.314Z","127.0.0.1" "de62ccdc-1e42-403c-a047-3676812219cc","security","login","False","success","geladm","SASLogon","2023-06-13T09:22:12.019Z","127.0.0.1" "84f530bf-4f93-4b78-893e-e200bf0cba68","security","login","False","success","geladm","SASLogon","2023-06-19T02:53:41.367Z","127.0.0.1" "be704891-ac56-4646-bd89-18206597a4ab","security","SessionAuthenticationSuccess","False","success","geladm","SASLogon","2023-06-19T02:53:41.375Z","127.0.0.1" "fbad594a-8571-4e32-83b3-95204d90d930","security","AdministrativeAction","True","success","geladm","SASLogon","2023-06-19T02:53:48.372Z","127.0.0.1" "e8556c28-e72d-4f3f-b281-207f12f15f68","security","login","False","success","geladm","SASLogon","2023-06-19T03:22:09.338Z","127.0.0.1" "4591fdf7-25c6-4ea2-b705-ab60e89b5fbf","security","SessionDestroyed","True","success","geladm","SASLogon","2023-06-19T03:24:38.374Z","127.0.0.1" "c8005e77-0250-4287-bd45-5eda35238687","security","login","False","success","geladm","SASLogon","2023-06-19T04:28:44.194Z","127.0.0.1" "7a012678-70ae-424c-9088-c5890e9a2d7c","security","SessionAuthenticationSuccess","False","success","geladm","SASLogon","2023-06-19T04:28:44.198Z","127.0.0.1" "d85ebff4-74e4-4762-8b3c-3802a43a2504","security","AdministrativeAction","True","success","geladm","SASLogon","2023-06-19T04:28:45.906Z","127.0.0.1" "aa407dcb-50fb-4951-86ca-5e39c9db5989","security","login","False","success","Ahmed","SASLogon","2023-06-19T04:31:02.689Z","127.0.0.1" "15a68ff7-a6cc-4b18-a8df-be770269cf61","security","SessionAuthenticationSuccess","False","success","Ahmed","SASLogon","2023-06-19T04:31:02.691Z","127.0.0.1" "4c8f6b3b-32ca-4f00-b3fe-4d4a482624ad","security","SessionDestroyed","False","success","Ahmed","SASLogon","2023-06-19T04:37:00.375Z","127.0.0.1" "cf42e5fa-ea42-430d-b8a7-c095be8c9194","security","login","False","success","Delilah","SASLogon","2023-06-19T04:37:05.142Z","127.0.0.1" "17404791-57b0-4f62-9c1d-5c763ec4ceda","security","SessionAuthenticationSuccess","False","success","Delilah","SASLogon","2023-06-19T04:37:05.149Z","127.0.0.1" "50d7eb6e-89d9-42f3-817a-591dde1ca3af","security","SessionDestroyed","True","success","geladm","SASLogon","2023-06-19T04:59:38.474Z","127.0.0.1" "6a44bbf8-b0a3-4171-9e02-337364bdeb6e","security","login","False","success","Delilah","SASLogon","2023-06-19T05:22:14.95Z","unknown" "1af2d753-5235-4cdf-b754-bc5397d2661e","security","SessionDestroyed","False","success","Delilah","SASLogon","2023-06-19T05:32:38.52Z","127.0.0.1"   As the sample output above shows, activity records are currently capturing predominantly security-type records. This is expected to grow in future releases to include records from more and more applications for capturing additional types of user actions.   You can also add additional flags for filtering the output. For example, you can narrow the results by user, application, action, or time range. Consult the built-in documentation to view all options: getactivityrecords.py -h   Output:   usage: getactivityrecords.py [-h] [-l LIMIT] [-t TYPE] [-a APPLICATION] [-c ACTION] [-d ADMIN_ACTION] [-s STATE] [-u USER] [-A AFTER] [-B BEFORE] [-S SORTBY] [-o {csv,json,simple,simplejson}] optional arguments: -h, --help show this help message and exit -l LIMIT, --limit LIMIT Maximum number of records to display -t TYPE, --type TYPE Filter by entry Type -a APPLICATION, --application APPLICATION Filter by entry Application -c ACTION, --action ACTION Filter by entry Action -d ADMIN_ACTION, --admin-action ADMIN_ACTION Filter by Administrative Action -s STATE, --state STATE Filter by entry State -u USER, --user USER Filter by Username -A AFTER, --after AFTER Filter entries that are created after the specified timestamp. For example: 2020-01-03 or 2020-01-03T18:15Z -B BEFORE, --before BEFORE Filter entries that are created before the specified timestamp. For example: 2020-01-03 or 2020-01-03T18:15Z -S SORTBY, --sortby SORTBY Sort the output ascending by this field -o {csv,json,simple,simplejson}, --output {csv,json,simple,simplejson} Output Style   Understanding how your SAS Viya environment is being used can help SAS Viya administrators to meet regulatory obligations, monitor and review security aspects, troubleshoot effectively and better align platform usage with broader business objectives. By leveraging the power of Python and the flexibility of SAS Viya's REST API, getactivityrecords.py enables the retrieval and analysis of activity records to provide valuable insights into user patterns for administrators and auditors.     Find more articles from SAS Global Enablement and Learning here. ... View more

In today's fast-paced world of infrastructure management, monitoring and alerting systems play a vital role in ensuring the health and performance of applications and services. An earlier post introduced the alerting capabilities provided by Prometheus for SAS Viya, which help administrators identify anomalies and respond to incidents proactively. This refresher article provides a high-level recap of what is available out of the box, what it can help with, and covers the process of creating new, custom alerts to enhance real-time monitoring.   Alerting is the process of triggering actions when defined conditions are met. In a SAS Viya deployment (with SAS Viya Monitoring for Kubernetes deployed), those conditions are based on metric information collected by Prometheus (about health and performance of your SAS Viya deployment) or log data collected by OpenSearch.   Alerts can help administrators proactively detect and predict potential anomalies with the deployment and/or infrastructure. When a defined alert condition or threshold is met, Prometheus lets the companion Alertmanager module know. Alertmanager can send notifications to a variety of channels, as well as take other actions automatically when an alert is triggered (such as creating a JIRA or ServiceNow ticket, or even autoscaling up/down). It also provides a facility for managing the lifecycle of triggered alerts (for example, to supress alerts while they’re being investigated).   So what kind of conditions can trigger alerts? When Prometheus is deployed, it includes a heap of generally useful alerts for common conditions out-of-the-box. These include alerts for scenarios like pods stuck in a CrashLoopBackOff state, StatefulSets not matching the expected number of replicas, Persistent Volumes filling up, or elevated CPU throttling. But administrators can also supplement these with their own bespoke alerts.   Before we get to the "how", let’s look at the type of metric information available in Prometheus. This can be done by submitting PromQL (the native, SQL-based querying language) expressions in the Prometheus Expression Browser web application. Conveniently, the Expression Browser includes a metrics explorer and has auto-complete, so you can begin by just selecting a metric and viewing the data available, and then adjusting your expression to filter for what you’re interested in seeing (and alerting on).   For example, assume we want to see memory usage for the sas-cas-control pod. If the word “memory” is typed into the query box, matching metrics are displayed and can be selected from the drop-down box. The container_memory_usage_bytes metric looks like a good match for our use case. A filter can be added by appending a curly brace to the query. We want to filter by pod, so when we start typing pod=" , auto-complete again lets us click on the desired pod name from a drop-down list.   In this example, our resulting expression is container_memory_usage_bytes{pod="sas-cas-control-6c9455dd78-g58fr"} . If we hit the Execute button, we get the metric data values (in bytes) for the query.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   Assume we want to create an alert for this metric. For instance, say we want an alert to be triggered when the sas-cas-control pod exceeds 80% of its memory limit, which we can find by running a kubectl describe on the pod (or for bonus points, try using the Expression Browser to find the appropriate metric and filters that will also give you the limit). Our expression would then become something like [memory usage] / [memory limit] as a percentage, which translates to PromQL as: (sum by (pod) (container_memory_usage_bytes{pod="sas-cas-control-6c9455dd78-g58fr"})) / (sum by (pod) (kube_pod_container_resource_limits{pod="sas-cas-control-6c9455dd78-g58fr",resource="memory"})) * 100   We can then add an operator to our expression to define the alert threshold - if we want to alert the trigger when the usage reaches 80% of the limit, the expression becomes: (sum by (pod) (container_memory_usage_bytes{pod="sas-cas-control-6c9455dd78-g58fr"})) / (sum by (pod) (kube_pod_container_resource_limits{pod="sas-cas-control-6c9455dd78-g58fr",resource="memory"})) * 100 > 80   The process for creating the alert is outlined in an earlier post, but to recap, requires a new YAML file which defines the alert in a PrometheusRule (custom resource) definition to be created and submitted with kubectl:     tee ~/PrometheusRule.yaml > /dev/null << EOF apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: labels: prometheus: prometheus-v4m-prometheus-0 role: alert-rules name: prometheus-viya-rules namespace: monitoring spec: groups: - name: custom-viya-alerts rules: - alert: CASControlMemoryApproachingLimit annotations: description: sas-cas-control container memory usage is approaching its limit. This can result in OOM errors and CAS sessions closing unexpectedly. summary: CAS Control high memory usage expr: (sum by (pod) (container_memory_usage_bytes{pod="sas-cas-control-6c9455dd78-g58fr"})) / (sum by (pod) (kube_pod_container_resource_limits{pod="sas-cas-control-6c9455dd78-g58fr",resource="memory"})) * 100 > 80 labels: severity: critical EOF kubectl -n monitoring create --filename ./PrometheusRule.yaml       Once created, the alert will be visible in the Prometheus UI's Alerts page:     If it when it triggers, it will turn red, and will also appear in the separate Alertmanager UI. When alert conditions are addressed/remediated, the alerts automatically return to a state of “Inactive” (not firing) without any intervention from an administrator. Administrators can "silence" firing alerts in the Alertmanager UI as a way of acknowledging them while they are being investigated.   Obviously, admins still need to specify what happens when the alert does trigger – for example, deciding who gets notified and how. This requires modifying the Alertmanager configuration. A good starting point is this post on alert routing. There are several earlier articles which outline how to configure Alertmanager to integrate with various third-party applications such as MS Teams, and Slack. . The command-line based amtool can also be very useful here.   Leveraging the powerful and flexible alerting capabilities provided by Prometheus in SAS Viya can enable organizations to stay ahead of critical situations, respond promptly to incidents, and optimize their operational efficiency.   For additional information, refer to the official Prometheus documentation and the SAS documentation on SAS Viya’s logging and monitoring. ... View more

Update to post: The open source configuration is available to SAS compute and batch sessions as of 2024.07, along with new, simplified documentation consolidating the instructions for configuring Python integration.   One of the key features of SAS Viya is its ability to integrate with different programming languages, including Python. SAS users can submit Python code for execution in SAS Viya with the PYTHON procedure; check out this blog post for an overview of what's possible. Some configuration changes are required to get set up, after which users can start running Python code from a SAS Compute session. That's great for submitting from SAS Studio, but what about in batch?   The necessary setup is covered nicely by Scott McCauley in his article. When the steps are completed, users can submit Python code or programs for execution in a SAS Compute pod (e.g. in a SAS Studio session).   However, a peek inside the file site-config/sas-open-source-config/python/python-transformer.yaml suggests that these configuration changes will only apply to the "sas-compute-job-config" PodTemplate; that's the Custom Resources that tells Kubernetes how to initialise a SAS Compute pod. Any new SAS Compute pod that gets launched will contain these changes to support Python code execution. However, things work a little differently with batch submission of code. The sas-viya CLI has a batch plugin that allows SAS programs (which can contain a PROC PYTHON statement to embed Python code) to be submitted from the command line. With this method, the submitting user must specify a batch context (as opposed to a compute context) to use. By default, Viya is deployed with two batch contexts; default and default-cmd. Both of these use the "sas-batch-pod-template" PodTemplate to initialise a batch session to execute the request. Unfortunately, this template will not contain the changes made to the compute template that are necessary to run Python or R.   We can see this difference by comparing the the compute PodTemplate with the batch PodTemplate.   Running kubectl describe podtemplate sas-compute-job-config shows all the Python configuration, such as the required volume mounts, is present:   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   Those are missing in the output shown by running kubectl describe podtemplate sas-batch-pod-template . This means that when we try to run a SAS program containing Python code, it will fail, since none of the required configuration will exist in the pod launched to run the code.   The solution, then, is to modify the batch PodTemplate so that batch sessions also contain the required open source configuration.  We can just copy and paste the lines that already exist in the transformer for patching the compute PodTemplate to also patch the batch PodTemplate.   The lines of interest are towards the end of python-transformer.yaml:       apiVersion: builtin kind: PatchTransformer metadata: name: compute-job-python-transformer patch: |- # Add python volume - op: add path: /template/spec/volumes/- value: name: python-volume persistentVolumeClaim: claimName: sas-pyconfig # Add mount path for python - op: add path: /template/spec/containers/0/volumeMounts/- value: name: python-volume mountPath: /opt/sas/viya/home/sas-pyconfig readOnly: true # Add python-config configMap - op: add path: /template/spec/containers/0/envFrom/- value: configMapRef: name: sas-open-source-config-python target: kind: PodTemplate name: sas-compute-job-config version: v1       After copy and pasting the above section, and then editing to target the batch PodTemplate, the end of your python-transformer.yaml should now also include:       ... --- apiVersion: builtin kind: PatchTransformer metadata: name: enable-python-batch-transformer patch: |- # Add python volume - op: add path: /template/spec/volumes/- value: name: python-volume persistentVolumeClaim: claimName: sas-pyconfig # Add mount path for python - op: add path: /template/spec/containers/0/volumeMounts/- value: name: python-volume mountPath: /opt/sas/viya/home/sas-pyconfig readOnly: true # Add python-config configMap - op: add path: /template/spec/containers/0/envFrom/- value: configMapRef: name: sas-open-source-config-python target: kind: PodTemplate name: sas-batch-pod-template version: v1       If you've done all the earlier pre-req steps, your kustomization.yaml file won't require any further changes, so you can just build and apply (or deploy your sasdeployment custom resource if you have the SAS Deployment Operator configured).   Now let's verify the changes. When deployment completes, run kubectl describe podtemplate sas-batch-pod-template|grep -A5 -B5 sas-pyconfig to check that the batch PodTemplate has been updated:       k:{"mountPath":"/opt/sas/viya/home/commonfiles"}: .: f:mountPath: f:name: f:readOnly: k:{"mountPath":"/opt/sas/viya/home/sas-pyconfig"}: .: f:mountPath: f:name: f:readOnly: k:{"mountPath":"/sashelp"}: -- Mount Path: /opt/sas/viya/config/etc/SASSecurityCertificateFramework/private Name: security Sub Path: private Mount Path: /gelcontent Name: sas-viya-gelcorp-volume Mount Path: /opt/sas/viya/home/sas-pyconfig Name: python-volume Read Only: true Dns Policy: ClusterFirst Image Pull Secrets: Name: sas-image-pull-secrets-k95ckd46h8 -- Nfs: Path: /shared/gelcontent Server: pdcesx02072.race.sas.com Name: python-volume Persistent Volume Claim: Claim Name: sas-pyconfig Events:       Much better. Then the final test; checking to see if batch submission works.   The program we want to run looks like:       proc python; submit; print('hello world') endsubmit; run;       The command that runs it looks like: sas-viya batch jobs submit-pgm --pgm-path /shared/gelcontent/gelcorp/shared/code/my_code.sas --context default --watch-output --wait-log-list --results-dir ~/   And that results in:       >>> The file set "JOB_20230406_102344_048_1" was created. >>> Uploading "test.sas". >>> The job "1dda5231-1978-4ab0-9da4-d0e264fdecde" was submitted. >>> Waiting for the job to complete... >>> Job started. 1 The SAS System Thursday, April 6, 2023 10:23:00 AM NOTE: Copyright (c) 2016 by SAS Institute Inc., Cary, NC, USA. NOTE: SAS (r) Proprietary Software V.04.00 (TS M0 MBCS3170) Licensed to THIS ORDER IS FOR SAS INTERNAL USE ONLY (SIMPLE), Site 00000000. NOTE: This session is executing on the Linux 3.10.0-1062.12.1.el7.x86_64 (LIN X64) platform. NOTE: Additional host information: Linux LIN X64 3.10.0-1062.12.1.el7.x86_64 #1 SMP Tue Feb 4 23:02:59 UTC 2020 x86_64 Red Hat Enterprise Linux release 8.7 (Ootpa) NOTE: SAS initialization used: real time 0.02 seconds cpu time 0.02 seconds NOTE: AUTOEXEC processing beginning; file is /opt/sas/viya/config/etc/sasgrid/default/autoexec.sas. NOTE: DATA statement used (Total process time): real time 0.00 seconds cpu time 0.00 seconds NOTE: AUTOEXEC processing completed. NOTE: The LOCKDOWN option has been set. SAS is now in the lockdown state. NOTE: The LOCKDOWN option has been set. SAS is now in the lockdown state. 1 proc python; 2 submit NOTE: Python initialized. Python 3.8.13 (default, Apr 4 2023, 04:53:36) [GCC 8.5.0 20210514 (Red Hat 8.5.0-16)] on linux Type "help", "copyright", "credits" or "license" for more information. >>> >>> 2 ! ; 3 print('hello world') 4 endsubmit; 5 run; 6 %let SAS_workpath = %sysfunc(pathname(work)); >>> hello world >>> NOTE: PROCEDURE PYTHON used (Total process time): real time 1.36 seconds cpu time 0.01 seconds NOTE: SAS Institute Inc., SAS Campus Drive, Cary, NC USA 27513-2414 NOTE: The SAS System used: real time 1.40 seconds cpu time 0.05 seconds 2 The SAS System Thursday, April 6, 2023 10:23:00 AM <<< Job ended. <<< Downloading the files in the file set to the following path: "/home/cloud-user//JOB_20230406_102344_048_1". <<< Downloading "test.log". <<< Downloading "test.sas". <<< Copying the log and list files from "/home/cloud-user//JOB_20230406_102344_048_1" to the current directory. <<< Copying "test.log". <<< Deleting the results directory. <<< Deleting the batch job. <<< Deleting the file set.       Success!   There's probably a solid enough case for having the open source configuration applied to both the compute and batch PodTemplates out-of-the-box with the python-transformer.yaml; which will obviously simplify and reduce the setup tasks required. I will update this post with the revised steps if the default setup changes.  Update: The open source configuration is available to SAS compute and batch sessions as of 2024.07, along with new, simplified documentation consolidating the instructions for configuring Python integration.   Thanks for reading. My thanks also to my colleaque @Bogdan_Teleuca for his contributions.   Find more articles from SAS Global Enablement and Learning here. ... View more

Effective alerting requires that the right people are notified of issues at the right time and via the most appropriate channels. SAS Viya Administrators and DevOps engineers using Alertmanager to manage alerts in the SAS Viya platform can integrate Alertmanager with several third party party applications to distribute alert notifications. While native support is built-in for some applications, many others are capable of receiving information via webhooks (e.g. alert notifications sent by Alertmanager). In this post, we'll look at how Alertmanager can integrate with Microsoft Teams via webhook, allowing alert notifications to be automatically posted to a Microsoft Teams channel when alerts are triggered.   There are several ways to set up the integration. In any case, as there is no native support for Teams integration in Alertmanager, and because Teams requires messages to be in a certain format (one that Alertmanager doesn't produce its alerts in), a third-party tool must be deployed to act as an intermediary. There are many to choose from, but in this case I used an open-source utility called prometheus-msteams. It is essentially a lightweight web server which receives alert messages from Alertmanager and sends them to a Teams channel in the required format.   But first, the Teams channel must be configured. Generate the unique incoming webhook URL (which is where prometheus-msteams will send messages) in the desired Teams channel by right-clicking on the channel name and clicking Connectors.     Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.    If it's not already added, click Add on the Incoming Webhook Connector. When it's added, click Configure.        Provide a name and then click create to generate the unique URL. Copy/take note of this, as it is required later.         Next, set up prometheus-msteams. There are several ways to deploy it, including as a binary or as a docker container. In my lab, I deployed it to my Kubernetes cluster (in the monitoring namespace deployed by SAS Viya Monitoring for Kubernetes) using the Helm chart. First, I added the chart repo as per the instructions with:   helm repo add prometheus-msteams https://prometheus-msteams.github.io/prometheus-msteams/   Then prepare the configuration file before deployment. This is where webhook URL from Teams must be specified.   tee ~/config.yaml > /dev/null << EOF replicaCount: 1 image: repository: quay.io/prometheusmsteams/prometheus-msteams tag: v1.5.1 connectors: # in alertmanager, this will be used as http://prometheus-msteams:2000/alertmanager – alertmanager: https://sasoffice365.webhook.office.com/webhookb2/e501090a-af3d-426c-bf35-2374d5e7d211@b1c14d5c-3625-45b3-a430-9552373a0c2f/IncomingWebhook/4b5a85a99819453aae87cbce67ea6469/07d2cf3f-f87b-4c4c-a3d6-2f10f2993322 # # Enable metrics for prometheus operator metrics: serviceMonitor: enabled: true additionalLabels: release: prometheus # change this accordingly scrapeInterval: 30s EOF   To deploy, run:   helm upgrade --install prometheus-msteams --namespace v4mmon -f config.yaml prometheus-msteams/prometheus-msteams   That results in:    Release "prometheus-msteams" does not exist. Installing it now. NAME: prometheus-msteams LAST DEPLOYED: Wed Mar 15 06:52:49 2023 NAMESPACE: v4mmon STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: ** Please be patient while the chart is being deployed ** To monitor the deployment, execute the following command: kubectl get pods -l app=prometheus-msteams --namespace v4mmon -w   Check that it has been deployed and is running:   kubectl get po -n v4mmon|grep msteams   The result:   prometheus-msteams-588777df8f-btmq2 1/1 Running 0 92s   The next step is to define a new receiver in the Alertmanager configuration for prometheus-msteams. In my lab, I made it the default receiver for all routes, so all alert notifications will get sent to Teams.   tee ~/alertmanager.yaml > /dev/null << EOF global: smtp_smarthost: $(hostname -f):1025 smtp_from: 'alertmanager@gelcorp.com' smtp_require_tls: false resolve_timeout: 5m route: receiver: prometheus-msteams group_wait: 30s group_interval: 5m repeat_interval: 12h receivers: - name: prometheus-msteams webhook_configs: - url: "http://10.96.3.151:1975/alertmanager" send_resolved: true EOF   Note that in this example, the URL is referring to the IP & port of the node where the prometheus-msteams pod is running. In my lab, I set up the prometheus-msteams service with a NodePort so that I could simulate test alerts using cURL (from outside the cluster).   Apply the updated configuration:   cat <<EOF | kubectl apply -f - apiVersion: v1 data: alertmanager.yaml: $(cat ~/alertmanager.yaml | base64 -w0) kind: Secret metadata: name: alertmanager-v4m-alertmanager namespace: v4mmon type: Opaque EOF   Once the Alertmanager config is applied, it may take a minute to take effect (hint: keep an eye on the Status page in the Alertmanager UI). When it does, notifications will start appearing as posts in the Teams channel when the alerts begin firing.        If you want to customize the way messages are posted, you can do that too by following the documented instructions and examples.   For more information refer to the GitHub page. Alternative approaches are also widely documented online.   If you'd rather use Slack than Teams, check out this previous post. Thanks for reading.   Find more articles from SAS Global Enablement and Learning here. ... View more

promtool is a command-line based utility delivered with Prometheus that performs some functions SAS and Kubernetes administrators will find useful. An earlier post demonstrated amtool, the equivalent CLI tool for Alertmanager. Where amtool focuses on verifying the Alertmanager configuration (including validation of alert notifications), promtool is geared towards querying and validating metrics and rules. When creating custom alerts for your Viya deployment, it can be particularly useful for testing the alert rules before you deploy them.   promtool is included as part of the Prometheus deployment, but can also be installed separately (requires golang) and run from anywhere. When SAS Viya Monitoring for Kubernetes is deployed, promtool can be found inside the Prometheus pod in the monitoring namespace at /bin/promtool.   Let's look at some examples of promtool in action.   Query metrics instantly from the command line   Run an instant PromQL query against your metric data, as you would in the Prometheus Expression Browser UI. The below example queries memory usage for the sas-visual-analytics container and displays the raw time-series metric data.   promtool query instant http://prometheus.gelcorp.com 'container_memory_usage_bytes{container="sas-visual-analytics"} / (1024 * 1024 * 1024)'   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   Query metric data over a range   Similar to above, but with values displayed over a specified period. For example, view the same data but in 30 second intervals over the last 5 minutes:   promtool query range http://prometheus.gelcorp.com 'container_memory_usage_bytes{container="sas-visual-analytics"} / (1024 * 1024 * 1024)' --step=30s --start=$(date -d '5 minutes ago' +'%s') --end=$(date -d 'now' +'%s')     This can be useful for determining where your alert thresholds should be.   Check alert and recording rules   Verify that your Prometheus rules are well-defined before deploying them. Put the rules in a rules file, and then use promtool to check them for syntax errors and misconfigured arguments. This also provides a nice way of validating automatic changes in a CI/CD environment. In this example, rules.yaml looks like:   groups: - name: custom-viya-alerts rules: - alert: ViyaMemoryUsage annotations: description: Total Viya 4 namespace memory usage is more than 20% of total memory capacity. summary: Alerting on Viya 4 namespace Memory usage runbook: https://gelgitlab.race.sas.com/GEL/workshops/PSGEL260-sas-viya-4.0.1-administration/-/blob/master/04_Observability/img/alerting/runbook.md expr: ((sum by (node) (container_memory_usage_bytes{container!~"POD",namespace="gelcorp",pod=~"sas-.+"})) / (sum by (node) (kube_node_status_capacity{resource="memory"})) * 100) > 20 labels: severity: critical - alert: LowAvailability expr: (avg_over_time(up{pod=~"sas-.+"}[1w]) * 100) < 70   The command to validate it:   promtool check rules ./rules.yaml       The result will either shows the number of rules detected or display any errors encountered with syntax or structure. Query labels   Find available values for a specified label name.   promtool query labels http://prometheus.gelcorp.com 'severity'         Again, this one can also be helpful when creating alerts; specifically, when assigning labels to alerts for use when sending alert notifications. Unit testing of alerts   Another very nice feature provided by promtool is the ability to unit test rules. You can simulate conditions to verify that the (new) alerts would be triggered once deployed (e.g. conditions that have never happened before). The detailed steps are beyond the scope of this post (there are many examples available online, including in the offical doc), but the process requires you to create a test file and then pass it in as an argument to the 'test rules' subcommand.   And more...   There are several other functions available with promtool, including a command to verify the Prometheus configuration (useful if making changes). For a full list of options and examples, consult in the built-in documentation or the official Prometheus documentation.   Thanks for taking the time to read.    Find more articles from SAS Global Enablement and Learning here. ... View more

The SAS Deployment Operator makes light work of many deployment and administration tasks. My colleague Mike Goddard wrote a great introductory post about what it is, how it works, and deployment options. In this post, we'll look at the steps required to update the SAS Deployment Operator to keep it running smoothly with all the latest fixes and features.   Updating the updater?   Using the Deployment Operator is the recommended approach for applying updates (patches or version updates) to your SAS Viya deployment. In fact, if you don't have it deployed in your Viya environment, the recommended approach for applying Viya software updates is to first deploy it, and then use it to apply the updates. But what about updating the Deployment Operator itself? The Deployment Operator and the rest of your deployed Viya software can generally be updated independently from each other, and the Deployment Operator is backwards compatible with older versions of Viya. As a general recommendation, consider updating the Deployment Operator at least each time you update your Viya software to the latest version. Deploying the latest and greatest will help keep your environment well-oiled. In some cases, it is actually a requirement to upgrade the Deployment Operator as part of a version update to SAS Viya. The good news is that it is a relatively a straight-forward process!   Step 1: Back up and obtain new Deployment Assets   OK, so technically this is more than one step. The intial step requires you to get new Deployment Assets from the MySAS portal or using the viya4-orders CLI to replace the existing assets. The new assets contain the new transformers and manifests for the operator deployment.   First, back up the existing $operator-deploy directory (in this example, the operator is deployed in cluster-wide mode😞   cp -r /home/cloud-user/project/deploy/operator-deploy /home/cloud-user/project/deploy/operator-deploy-orig   Then delete the operator's existing/old sas-bases directory:   cd /home/cloud-user/project/deploy/operator-deploy rm -rf sas-bases   After getting the new assets (for the later/latest version), extract the tarball to create a new sas-bases directory.   tar xvfz SASViyaV4_9CKY1V_stable_2022.09_20221010.1665437790467_deploymentAssets_2022-10-10T224758.tgz Step 2: Copy new operator files   Copy the required operator files to the top-level $operator-deploy directory and make them writable.   cd /home/cloud-user/project/deploy/operator-deploy sudo cp -r sas-bases/examples/deployment-operator/deploy/* . chmod +w site-config/transformer.yaml   Step 3: Restore YAMLs from backups   Rather than making required updates to the kustomization.yaml and transformer.yaml files manually, restore the files from the backups taken earlier, which already contains the necessary customisations such as setting required ClusterRoleBindings, configuring a mirror registry and choosing between cluster-wide mode and namespace mode.   # restore from backups cp ../operator-deploy-orig/site-config/transformer.yaml site-config/ cp ../operator-deploy-orig/kustomization.yaml .   If necessary, make any additional configuration changes after restoring. Be sure to glance over the deployment instructions to verify whether any new customisations are requried. If you repeat this process the next time you update the operator, new additions will be retained next time around.   Step 4: Deploy   And the last step is to deploy. In cluster-wide mode, specify the name of the operator's namespace.   kustomize build . | kubectl -n sasoperator apply -f -   As demonstrated, the process is essentially a redeployment using new assets (but note that fields in transformers.yaml refer to existing values). No clean up is necessary before or after the update, and the operator should not be stopped prior to commencing the update. The new Deployment Operator will be available once the operator pod starts after you deploy (in the last step). Be sure to read the documentation just in case any additional changes are required to transformer.yaml or kustomization.yaml after restoring them.   Thanks for reading.     ... View more

An earlier post outlined the steps involved in setting up archiving for older audit records out of SAS Viya's Audit service. A new feature has since been added to SAS Viya to optionally enable the purging of these archived audit records once they reach a defined retention period to keep the PV storage clean and tidy. This post demonstrates the required steps.   There are two relevant configuration instances to set up the purging of archived audit records. The first, sas.audit.archive, was reviewed in the previous post, but now contains some additional configurable properties.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.     The *.retention.in.archive properties control the period for which audit and activity records are kept in the archive destination before they are purged. Define your retention period in these fields when setting up archiving. If your customer requires records to be kept for longer, adjust the value as necessary. In the example above, archived audit records more than a day old will be cleaned up by the purge process. Also note that another new property is also now available to define the minimum amount of available space required on storage before archiving occurs. Set your threshold value (a percentage of available space) in the storage.local.remainingSpaceThreshold property as required.   Then there's the new sas.audit.purge configuration instance which toggles the purge function on and off, and defines the frequency at which the purge process will run.       To enable purging, ensure that the enabled switch is turned on, and then define the schedule using cron standard syntax. In the same way that audit records are archived to the attached storage volume according to the schedule defined in the scanSchedule property, the purge process will remove record from the defined archive volume path per the schedule.   If backups of archived audit records (or PVCs) are required, be sure to take them between scheduled purges. Admins can also opt to disable the purge facility and manually delete archived records on an ad-hoc basis, but elevated privileges are required.   Thanks for reading. For more on the Audit service, refer to the SAS Viya: Administration guide.   Find more articles from SAS Global Enablement and Learning here. ... View more

Anyone who’s tried modifying the Alertmanager configuration knows how tedious a process it can be. For one, indentation in the configuration.yaml (or alertmanager.yaml) is important, and very little feedback is provided by Kubernetes or Alertmanager if something is invalid or incorrect. As a result, it can be tricky to determine whether the updates have actually taken effect. When the kubectl command to apply the new configuration is run, a message indicates the configuration was updated. But if there are problems with the configuration, such as indentation or syntax issues, they don’t become apparent until the administrator sees that configuration hasn’t actually been updated (by looking the Alertmanager UI, or worse; by realising that alerts aren’t behaving as they should). Even then, it can be difficult to identify exactly why the configuration wasn’t updated. Where is the invalid element or misplaced line break? Introducing amtool, Alertmanager’s command-line companion that provides, among several other neat functions for interacting with alerts, a way for the configuration to be validated to quickly identify issues.   amtool is a CLI tool bundled with Alertmanager. In SAS Viya Monitoring for Kubernetes, it is deployed and accessible from inside the Alertmanager pod with a command like: kubectl -n v4mmon exec -it alertmanager-v4m-alertmanager-0 -- amtool [command]).   This may not be the most convenient way to run it though, as there are some caveats, of which an obvious one is the requirement for direct access to the cluster (and there are others). Alternatively, if you set up Ingress for Alertmanager (or a NodePort for the service), you can install the amtool CLI anywhere, or even run it as a standalone Docker container. In my lab, I cloned the Prometheus repo and compiled the amtool binary per the instructions (requires Golang to be installed).   Back to our primary use case. Assume we want to update the Alertmanger configuration to define a new receiver. First, let's use amtool to query the existing configuration: amtool --alertmanager.url=http://alertmanager.server.gelcorp.com/ config show   global: resolve_timeout: 5m http_config: follow_redirects: true smtp_hello: localhost smtp_require_tls: true pagerduty_url: https://events.pagerduty.com/v2/enqueue opsgenie_api_url: https://api.opsgenie.com/ wechat_api_url: https://qyapi.weixin.qq.com/cgi-bin/ victorops_api_url: https://alert.victorops.com/integrations/generic/20131114/alert/ route: receiver: "null" group_by: - job continue: false routes: - receiver: "null" match: alertname: Watchdog continue: false group_wait: 30s group_interval: 5m repeat_interval: 12h receivers: - name: "null" templates: - /etc/alertmanager/config/*.tmpl   Let's try adding a new route and receiver by loading a new configuration file (alertmanager.yaml), which looks like:   global: smtp_smarthost: mail.gelcorp.com:1025 smtp_from: 'alertmanager@gelcorp.com' smtp_require_tls: false resolve_timeout: 5m route: receiver: sas-email-alert group_wait: 30s group_interval: 5m repeat_interval: 12h receivers: - name: sas-email-alert email_configs: - to: sasadmin@gelcorp.com headers: Subject: 'New Alert - SAS Viya' send_resolved: true require_tls: false   Create a new YAML manifest to update the secret which stores the base64-encoded Alertmanager configuration, and apply it:   tee ~/alertmanager-secret.yaml > /dev/null << EOF apiVersion: v1 data: alertmanager.yaml: $(echo $(cat ~/alertmanager.yaml | base64 -w0)) kind: Secret metadata: name: alertmanager-v4m-alertmanager namespace: v4mmon type: Opaque EOF kubectl apply -f ~/alertmanager-secret.yaml -n v4mmon   In the output, we get the following message:     Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.       But a look at the Alertmanager configuration shows the old configuration, which doesn’t include our updates. Despite the message printed to the terminal, the new configuration was not applied.   Here’s where amtool can help us troubleshoot. If we run the below command, we can figure out what’s broken: amtool --alertmanager.url=http://alertmanager.server.gelcorp.com/ check-config alertmanager.yaml   The culprit: line 13. Sure enough, the email_configs block has an invalid element; the "to" field is incorrectly indented.       After adding a space to correct it, we can try checking it again. This time, it looks more positive:       This is the config file we should apply to ensure our Alertmanager configuration update is successful.   There are many other useful functions provided by amtool, including:   Displaying alert routes: amtool config routes show   Testing alert routes (in this example, to see where an alert with a specified label will be sent to): amtool config routes test severity=warning Viewing alerts: amtool alert as well as a bunch of other cool stuff like managing silences and testing new alerts.   amtool provides a simple, quick and effective method for managing Alertmanager configuration and artifacts. Instructions can be scripted and scheduled, and very little setup is required to get started. The built-in documentation contains a comprehensive list of possible commands available to the administrators (hint: you can also enable bash auto-completion with: amtool --completion-script-bash ).   Thanks for reading.   Ajmal Farzam     Find more articles from SAS Global Enablement and Learning here. ... View more