In the world of monitoring and observability, the ability to detect and respond to issues promptly is paramount. Alertmanager and Grafana, both deployed with SAS Viya Monitoring for Kubernetes, possess alerting capabilities. While both serve the purpose of alerting, they have distinct features and approaches that cater to different needs. In this blog post, we'll endeavour to compare Alertmanager and Grafana by their alerting functions and provide some points to consider for when you might want to use one over the other.   Alertmanager's key features   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   Alertmanager is an open-source alerting tool that works seamlessly with Prometheus for a comprehensive metric-based monitoring and alerting toolkit. It is designed to handle alerts sent by client applications such as Prometheus and then manage these alerts by grouping, deduplicating, and routing them to various integrations like email, PagerDuty, Slack, and Microsoft Teams. Both are deployed together with SAS Viya Monitoring for Kubernetes. Since Alertmanager is tightly integrated with Prometheus, it leverages PromQL, Prometheus' powerful query language for defining alerting rules. This makes it easy to create sophisticated alerting conditions based on Prometheus metrics.   Alerts are created as PrometheusRules, and evaluated against metric data stored in Prometheus. Rules use YAML; easy to use, edit, store and deploy. Alertmanager allows users to define complex alert routing and inhibition rules based on labels attached to alerts. This flexibility enables fine-grained control over which alerts are sent to which receivers. For more details on how the routing process works, check out thisearlier post. Alertmanager provides a silence mechanism that allows users to temporarily mute specific alerts or entire groups of alerts.; useful when dealing with issues and addressing underlying causes. Alerts can also be inhibited in advance during planned outages, such as during maintenance activities. Alertmanager can be deployed in a highly available configuration to ensure reliability. It supports clustering and replication for fault tolerance.   Grafana's key features     Grafana is a popular open-source platform for monitoring and observability that offers a wide range of features, including visualization, dashboarding, and alerting. While Grafana is primarily known for its visualization capabilities, it also provides robust alerting functions. Older versions of Grafana required alerts to be created on dashboards, which was quite restrictive. While this is still possible, the Alerting facility that is now included offers much more comprehensive and flexible capabilities.   Grafana allows users to define alerting rules visually using its intuitive web interface. Users can set thresholds, conditions, and notification channels for triggering alerts directly in the UI (no need to write YAML or run kubectl commands). Grafana's alerting feature seamlessly integrates with its visualization capabilities. Users can create alerts directly from dashboards and visualize alerting states alongside other metrics. Grafana supports a variety of data sources; not just Prometheus. This flexibility enables users to create alerts based on data from different monitoring systems. Useful if other existing tools are already in use. Grafana keeps a history of triggered alerts, making it easy to track the lifecycle of alerts over time. Additionally, users can annotate graphs with alert events for better context.   Comparison   Authentication   Grafana requires users to log in with a username and password. Alertmanager has no such authentication built in; if the URL is accessible, anyone can use it. There are other ways to 'control' access to Alertmanager, but they need careful planning and consideration. Here is one example. Certainly, none are as convenient as Grafana's simple login prompt.   Ease of Use   Grafana's web interface provides an intuitive way to create and manage alerts (visually), making it more accessible to users who are not familiar with Prometheus' query language. On the other hand, Alertmanager's configuration may require more expertise due to its reliance on Prometheus' query language.     Default alerts   Prometheus includes a number of pre-defined alerts for Kubernetes applications when SAS Viya Monitoring for Kubernetes is deployed. Very useful for alerting on generally common issues, and no additional setup is required beyond defining notification channels for them. Grafana conveniently also displays these alerts, but interacting with them (e.g. silencing them when they fire) is a little less intuitive for these than it is for Grafana-managed alerts.     Integration   Both AlertManager and Grafana offer integration with various notification channels such as email, Slack, and PagerDuty. However, Grafana's integration capabilities extend beyond alerting to include visualization and dashboarding (Grafana is first a monitoring tool after all). There are also options for integrating the two together; for instance, to send Grafana-managed alerts to Alertmanager when they begin firing.   Scalability   Alertmanager can easily be scaled horizontally for high availability to handle large volumes of alerts; in fact, it's designed to be clustered. Grafana's scalability depends on the underlying data source it's connected to, but it can also scaled with a bit of effort. With SAS Viya Monitoring for Kubernetes, though, both are deployed as standalone instances by default.   Customisation   Alertmanager provides more fine-grained control over alert routing and inhibition rules, making it suitable for complex alerting workflows. It also offers options to customise nearly every aspect of the alert notifications. Grafana, on the other hand, offers greater flexibility in terms of visualization and dashboard customization, which Alertmanager lacks.   Conclusion   Both AlertManager and Grafana are powerful tools for alerting in the context of monitoring and observability. The choice between the two (or both!) depends on factors such as familiarity with Prometheus, integration and security requirements, and needs for advanced alerting features. Grafana may generally be considered more convenient for administrators with basic alerting requirements or whom are new to observability tools, whereas the expertise required to best leverage Alertmanager's powerful capabilities may be better suited to organisations that already use and have experience with it and related tools (e.g. Prometheus). Ultimately, organisations should evaluate their specific use cases and requirements to determine which tool best fits their needs. ... View more

The event-driven architecture in SAS Viya 3.5 (and 3.4) offers the capability for effective auditing of system resources, report activity, data access, and more. In this post, we’ll revisit the options available for auditing user access and highlight important caveats and considerations. This revised version of an earlier article has been updated to account for some changes in the underlying processes to provide a clearer picture.   CAUTION: Important considerations   Before adopting the approach to auditing user sessions documented in this post, beware these important cautions about using the EV Datamart data captured by the SAS Operations Infrastructure for this purpose: The data is not originally intended as 'audit' and it does not contain data integrity and security features that true audit data should. The data is derived from different data sources in which information is recorded differently, and matching user sessions with this data can be imprecise. It is not suitable for real-time or near-real-time purposes due to (potentially significant) latency in data being written to disk. Matching session activity like logons and logoffs using the session ID may not always work as expected. While the session ID is generally unique, it is not guaranteed to be unique. There are a small number of situations and edge-cases where a given session ID might be reused. SAS Viya 3.5 versions released in December 2023 (and later) address an issue where the session signature was not recorded for a login event, resulting in errors with the AUTHENTICATIONS table data. The information outlined in this post is a potential workaround for simplifying the process for auditing user session, but the caveats listed above must be noted as they present some potential challenges with this approach. Proceed with caution.   Information about logins and user sessions is captured by the Audit service in the AUDIT.security_audit tables in the SAS Infrastructure Data Server. The default behaviour, as defined in the security type properties in the sas.audit.record configuration instance, is to capture read failures only on security actions.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.    When trying to audit user login attempts, the login event provides information about (successful and failed) login attempts coming from the SASLogon application.   Let’s look at an example. Below, we use the sas-admin CLI’s audit plugin to see that user Ahmed logged in at 07:31 AM on the 25th of October (note that the timestamp is in UTC in PostgreSQL, but is converted to local time when a SAS Operations Infrastructure ETL job loads it to the AUDIT table in CAS).   /opt/sas/viya/home/bin/sas-admin --output text audit list --user-id Ahmed --sort-by ~timeStamp   ID Time Stamp Action State User ID Application URI c72c4533-886d-493b-9ef2-1dcff4ff4674 2019-10-25T07:57:54.776Z SessionDestroyed success Ahmed SASLogon bb748c54-6cb3-4edf-adbe-fce516c5786c 2019-10-25T07:31:21.014Z update success Ahmed reportImages /reportImages/jobs/4d7a64d8-1839-4d5b-a562-b140cdfc1f22/state caf1264d-4d37-444b-af45-4e339edc5dc6 2019-10-25T07:31:20.999Z create success Ahmed reportRenderer /reportRenderer/images 713e18ac-34db-4115-92aa-35f7662b0909 2019-10-25T07:31:20.649Z read success Ahmed reportData /reportData/commons/settings c5df6523-eb4f-42fe-b89b-fbdcc73d1c56 2019-10-25T07:31:20.298Z create success Ahmed reportPackages /reportPackages/jobs/b7325111-3f5a-4659-84bf-d1e59bdfbc74 e8578d9f-73b1-4dea-a94e-048a9eb9a72d 2019-10-25T07:31:20.030Z create success Ahmed reportData /reportData/jobs/61c55ce8-2b0a-4989-a14f-e1a5395b3b86 5aec043f-c2f8-46b6-a758-bc040abf1f69 2019-10-25T07:31:17.990Z create success Ahmed reportData /reportData/metadataSupplements fc7c64d8-bf75-40f5-a851-4afa5dc29beb 2019-10-25T07:31:16.141Z create success Ahmed reportImages /reportImages/jobs/4d7a64d8-1839-4d5b-a562-b140cdfc1f22 f9ad69f9-22b8-447b-82c2-45ebeb462409 2019-10-25T07:31:14.523Z read success Ahmed reportData /reportData/commons/settings 5dc96422-833c-496c-bd65-64659faf76d0 2019-10-25T07:31:08.641Z read failure Ahmed preferences /preferences/preferences/@currentUser/FormatLocale.DefaultLocale 5da5acf5-4d6b-47b3-9da5-c30ea7c3d946 2019-10-25T07:31:08.096Z read failure Ahmed preferences /preferences/preferences/@currentUser/_hc_xx_ 4a4c470f-afe9-469c-af3a-3fdb6d5f951e 2019-10-25T07:31:07.082Z read failure Ahmed preferences /preferences/preferences/@currentUser/FormatLocale.DefaultLocale b66f6eed-8ae8-47a3-8d3a-60e23df5cec9 2019-10-25T07:31:05.463Z login success Ahmed SASLogon   When a user logs off gracefully, a SessionDestroyed action is registered indicating the termination of the session as shown above.   If a user closes the browser instead of logging off, the SessionDestroyed does not get created in the audit records until the session times out. Session timeout duration is configurable.Set Time-out Interval for SAS Viya Web Applications in SAS® Viya® 3.5 Administration: Configuration ....   Some applications generate their own audit data when a user logs in. For example, SASStudio V creates a Compute server session when a user logs in, and an audit record similar to the following is captured.     /opt/sas/viya/home/bin/sas-admin audit list --user-id Ahmed --sort-by ~timeStamp --details --application compsrv   ID Time Stamp Type Action State Description User ID Application Remote Address URI 1967127d-87da-4656-922f-ed3b3bf1733d 2019-10-23T11:47:42.122Z resource start success 92ce6360-b23c-485b-94dc-48e5bd681bdd:https://intviya02.race.sas.com:38934/compute/servers/92ce6360-b23c-485b-94dc-48e5bd681bdd Ahmed compsrv   When the compute session is terminated, an audit record is captured indicating the deletion of the session (with the Description field indicating to the ID of the session).   /opt/sas/viya/home/bin/sas-admin audit list --user-id Ahmed --sort-by ~timeStamp --details --application compute   ID Time Stamp Type Action State Description User ID Application Remote Address URI 5ebd10ca-b4e1-4e7e-b71d-2874bfa672ca 2019-10-23T07:50:36.257Z resource delete success Deleted compute server 92ce6360-b23c-485b-94dc-48e5bd681bdd Ahmed compute /compute/servers/92ce6360-b23c-485b-94dc-48e5bd681bdd c9a1fc4a-48ea-4afa-a456-45e890571e42 2019-10-23T07:50:36.132Z resource delete success Deleted compute server session 92ce6360-b23c-485b-94dc-48e5bd681bdd-ses0000 Ahmed compute /compute/sessions/92ce6360-b23c-485b-94dc-48e5bd681bdd-ses0000   User sessions are assigned a session signature when they are established. The session signature (session_sig) property links the login event with the corresponding SessionDestroyed event. However, there are some important considerations.   First, refer to the caveats listed at the beginning of this post about potential issues with relying on the session signature for this purpose.   Second, the session signature is not visible when surfacing audit records using the CLI. This information is recorded in TSV files in the EV Datamart directory (/opt/sas/viya/config/var/lib/evmsvrops/evdm/application/security) on the Viya server. When a session is started:   2 34dca2cc-fcc9-4b6d-8c72-32d2958c9320 security application/vnd.sas.event.security 2019-10-15T06:21:18.973000+00:00 sasadm sas-deployment-id:dml5YQ==,sas-event-source:U0FTTG9nb24=,session_sig:NTNlZmNlZGE= security action:bG9naW4=,actionState:U1VDQ0VTUw==   Note the session_sig value. When the session ends, another record will be generated in the same file, referring to the original signature (orig_session_sig) for that session.   2 b41e8882-192d-4657-8fee-4cb04a96abda security application/vnd.sas.event.security 2019-10-15T07:02:30.282000+00:00 sasadm orig_session_sig:NTNlZmNlZGE=,sas-deployment-id:dml5YQ==,sas-event-source:U0FTTG9nb24=,session_sig:NDk0MGZmNw== security action:U2Vzc2lvbkRlc3Ryb3llZA==,actionState:U1VDQ0VTUw==   Fortunately, the legwork for extracting this information has already been done. Session information is extrapolated from these files and stored in the authentications subject table (opt/sas/viya/config/var/lib/evmsvrops/evdm/subjects/authentications.sas7bdat). Information is captured in this table for metering purposes, a secondary effect is that it provides a really nice way (with the caveats listed above!) to audit user session information. In the example below, several user sessions are listed, each with a datetime stamp of the login action, the session signature, and the duration of the session (in seconds).   Note that user Sasha’s session has a duration of 0.00 seconds. This points to an active user session (but there may be exceptions - listed above -  due to latency), indicating that Sasha has not yet logged out (SessionDestroyed action has not occurred for the session with this signature).   A job runs every 5 minutes to update this table, and another job runs daily to clear records older than 10 days. Update: To change the number of days (e.g. from 10 to 14 days) for which to capture records, execute:   /opt/sas/viya/home/bin/ops-dm-admin set EMI_DELETE_TSVZIP_DAYS=14   We can query this table to get some more insights about user sessions. Can you use it to figure out who was logged in at a certain time, for example?   A basic SAS program can be submitted to help answer that question.   /* Specify query time below in datetime. format */ %let qrytime='15OCT19:08:39:18'dt; libname evdm '/opt/sas/viya/config/var/lib/evmsvrops/evdm/subjects' access=readonly; data activesessions; set evdm.authentications; endtime=datetime+duration; format datetime datetime.; format endtime datetime.; where &qrytime between datetime and &endtime or (&qrytime >= datetime and (duration=0 and session_sig ne ''); run; proc print data=activesessions; run;   Note that this code sample is for demonstration purposes only, and there may be more efficient or more accurate methods to extract the required user session information.   Thanks to my colleagues Greg Smith, Carl Sommer, Fred Forst and Mike Roda for their contributions to this and the original version of this article, and thank you for reading! ... View more

PostgreSQL 15 is available in the November 2023 release of SAS Viya 3.5. SAS strongly recommends that customers upgrade their deployed PostgreSQL instance to version 15. Customers running SAS Viya 3.5 should check to see if they already have version 15 deployed, and if not, should consider upgrading in order to stay comfortably within PostgreSQL's support policy and to stay current with security fixes and patches. In this post, we will demonstrate how to check what version you are running in your environment, and highlight the steps you must perform to upgrade your deployed version of PostgreSQL to version 15.   If you are running a version of SAS Viya older than Viya 3.5, you must first upgrade your environment to SAS Viya 3.5 if you want to obtain PostgreSQL 15. Specifically, you should upgrade to at least Ship Event 23w44, which is the earliest release of SAS Viya 3.5 that contains PostgreSQL 15. This means that to get PostgreSQL 15, you must obtain a new order from SAS and then perform an upgrade. Any new deployments of SAS Viya 3.5 from 23w44 or later will deploy PostgreSQL version 15 by default. Similarly, if you are running a version of Viya 3.5 earlier than Ship Event 23w44, you need to upgrade to at least that release to get PostgreSQL 15. The simplest way to verify the version you are running is to directly query the version of PostgreSQL deployed in your environment by doing the following:   Determine the deployment target(s) for PostgreSQL in your environment In you inventory.ini file, look in the [sasdatasvrc] and [pgpoolc] (if applicable) host groups to identify the target machines, and then look in the [host_definitions] at the beginning of the file to view host information for those machines. Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   Check if the PostgreSQL 15 binaries have been deployed If SAS Viya is deployed on Linux, run the following command on the target machines: ls /opt/sas/viya/home/postgresql15 If not found, you do not yet have PostgreSQL 15 deployed.   Or if deployed on Windows, run the following in Powershell: dir "C:\Program Files\SAS\Viya\lib\postgresql15\bin"   If not found, you have not yet deployed PostgreSQL 15.   In these cases, you must contact SAS to obtain a new SAS Viya 3.5 software order from Ship Event 23w44 or later to upgrade to PostgreSQL 15. Once you have the new order, upgrade your SAS Viya deployment by following the documented instructions, remembering to perform the all-important pre-upgrade and post-upgrade steps for PostgreSQL. If you do have the PostgreSQL 15 binaries deployed, continue to the next step.   Verify whether your SAS Viya deployment is actually running the deployed PostgreSQL 15.   It is possible that your environment has PostgreSQL 15 deployed, but post-upgrade steps to configure it haven’t been performed. As an additional check, verify whether your SAS Viya deployment is actually running the deployed PostgreSQL 15.   On Linux, run the following as sudo on one of the target machines identified earlier: sudo cat /opt/sas/viya/config/data/sasdatasvrc/postgres/node0/PG_VERSION On Windows, run the following in Powershell:   cat C:\ProgramData\SAS\Viya\data\sasdatasvrc\postgres\node0\PG_VERSION A value of 15 listed in the PG_VERSION file indicates that PostgreSQL 15 is running and in use by SAS Viya. You can stop reading here, because all is as it should be!   However, if the listed version is not 15, some further action is required. We need to check to see if the latest PostgreSQL and related RPMs/MSIs are installed. On Linux, run the following to check the deployed RPM versions for PostgreSQL components:    cd /etc/init.d rpm -qa | egrep 'sasdata|sas-postgres|pool'   The versions displayed might resemble the below (the latter part of the values displayed for each RPM is irrelevant and has been omitted):   sas-sasdatasvrc-0.8.1-XXXXXXXXX.XXXXXXXX sas-postgresql-secure-libs-15-15.4-XXXXXXXXX.XXXXXXXX sas-pgpoolc-0.7.0-XXXXXXXXX.XXXXXXXX sas-postgresql-secure-15-15.4-XXXXXXXXX.XXXXXXXX sas-pgpool-II-44-4.4.4-XXXXXXXXX.XXXXXXXX If the versions shown above are displayed, this indicates you have deployed PostgreSQL 15 but not configured it yet. Some additional steps are required to reconfigure PostgreSQL and pgpool and complete the upgrade to version 15. Review the post-upgrade steps and completed all documented tasks.   If you do not have those versions, contact SAS to obtain a new SAS Viya software order from Ship Event 23w44 and upgrade your SAS Viya deployment in order to obtain PostgreSQL 15 Similarly, on Windows deployments, run the following in Powershell to check the MSI versions:    dir /powershell-deployment\Downloads   If sas-datasvrc15-15.4.0.0.msi   is listed, you have PostgreSQL15 deployed but not configured yet. Again, review the post-upgrade steps and completed all documented tasks. If you do not have that version, contact SAS to obtain a new SAS Viya software order from Ship Event 23w44 and upgrade your SAS Viya deployment in order to obtain PostgreSQL 15. A couple of additional points to consider:   Perform a full backup after updating your SAS Viya deployment After upgrading PostgreSQL, do not restore backups that were taken in older versions of PostgreSQL. If you have upgraded to PostgreSQL version 15, and you wish to then migrate to SAS Viya 4, you must move to a supported target version of SAS Viya 4 using an external instance of PostgreSQL 15.  Thank you for reading. My thanks also to my colleagues Cindy Taylor, Joseph Cho and Fred Perry for their contributions.        ... View more

Staying up-to-date with SAS Viya software is more important than ever. Updating regularly will give you the latest features and fixes, but also ensures that your deployment doesn't fall out of Standard Support. Before you set sail on your voyage to update to new SAS Viya version, it's essential to prepare properly to ensure a safe and smooth transition. In this article, we'll explore the important pre-upgrade tasks that will help reduce the risk of unforeseen issues and ensure you are set up for a successful update.   Pre-update Checklist   The SAS Viya Platform Administration documentation lists the SAS Viya platform versions that are in Standard Support and the recommended update paths to the latest version. In fact, the task of checking the support status of your deployment is the very first item in the Pre-update checklist of essential tasks that SAS now provides, which should be consulted before each update. If skipping versions or switching cadence as part of your update, take care to correctly identify interim versions which may be relevant; you will need to consult the Deployment Notes and execute the documented Before Deployment Commands for these.   In addition to checking your SAS Viya software support status, the checklist includes a task to verify your deployed Kubernetes version. Carefully read the documentation to see if you also need to upgrade to a supported version of Kubernetes or Kustomize.    Additionally, if your deployment includes it, consider updating the SAS Deployment Operator with each SAS Viya software update. Updates are backwards compatible, so running the latest version is a good practice. Note that if you have the SAS Deployment Operator deployed, you must use it to perform the SAS Viya update. If you used the sas-orchestration deploy method for the initial install, you must use that. It is not possible to mix and match deployment methods for update tasks.   A good practice is to run an inventory scan report before and after any major platform changes, including software updates. The two scans can be compared to verify that your content was not affected/changed during the update.   The inventory scan should also be run prior to a backup (and if required, after a restore). Pre-update backups should include an ad-hoc SAS Deployment Backup, backups of the $deploy directory and kustomization file, and if applicable, the SASDeployment CR file.   Next on the checklist is Deployment Assets. These should be retrieved for the target version you are moving to. Once downloaded, take a look at the README.md files for products you have deployed - this step is particularly important if you are adding new software as part of an upgrade, as there may be additional kustomizations or other manual steps that need to be performed for those new products. Note that assets automatically include the latest available license.   There are additional tasks in the checklist for multi-tenant environments and deployments that include SingleStore or multiple CAS servers. Be sure to work diligently through the list to ensure all relevant tasks are appropriately addressed.   Additional Tasks   Updates should be carefully co-ordinated with relevant technical resources to complete pre-requisite tasks. You may need to liaise and work with your Kubernetes administrator, security teams, change managers, end users, etc. to ensure you are ready for the update to begin. For example, you may need resources assigned to perform supplementary tasks ahead of time, such as:   providing sufficient access to enable you to perform the software update upgrading Kubernetes software (if required) verifying that the cluster has sufficient available resources to enable duplicates of each pod to run temporarily during the update notifying users of potential disruption updating the mirror registry, if applicable retrieving the output of the update checker job (bonus points for automating this)   Each organisation may have unique considerations, so seek further guidance from SAS Technical Support if necessary. With a well-executed pre-upgrade plan, you will avoid pitfalls and minimise disruptions on your way to the latest version of SAS Viya.     Find more articles from SAS Global Enablement and Learning here. ... View more

Grafana provides comprehensive insights into your SAS Viya deployment's health and performance using metrics collected by Prometheus. New capabilites were recently added to Grafana to improve its alerting capabilities. In this post, we will walk you through the process of configuring alerting for your SAS Viya deployment in Grafana.   Grafana requires Prometheus or another compatible data source set up to gather metrics from SAS Viya components. This is the default when SAS Viya Monitoring for Kubernetes is deployed.   The Grafana configuration is stored in a configMap (v4m-grafana) in the monitoring namespace. The config stored here includes things like your SMTP server host and port. If Grafana is already deployed, running a set of commands like the below will toggle on email connectivity and specify your mail server details in the configMap to allow Grafana to send alert notifications via email.   # get configmap in .yaml kubectl --namespace monitoring get configmap v4m-grafana --output=yaml > /tmp/v4m-grafana-configmap.yaml # update yaml file to enable smtp and enter mail server host:port sed -r -i "/kind: ConfigMap/ i\ \ \ \ [smtp] \\ \ \ \ enabled = true \\ \ \ \ \ host = mailserver.gelcorp.sas.com:25" /tmp/v4m-grafana-configmap.yaml # apply updated yaml kubectl -n monitoring replace --filename=/tmp/v4m-grafana-configmap.yaml # restart grafana kubectl -n ${_monitoringNS} delete po -l app.kubernetes.io/name=grafana   If Grafana isn't deployed, it's possible to specify these config parameters (and others) pre-deployment in the user.env file provided with SAS Viya Monitoring for Kubernetes.   Before actually getting to creating alerts, you must configure the notification channels to receive alerts. First, log on to Grafana and navigate to the "Alerting" section by clicking on the bell icon in the left sidebar.   Note that the main "Alert rules" page displays any other alerts configured in Prometheus. While there are limited options for interacting with these alerts, having the ability to view all alerts in one interface is very convenient. The best part is that when SAS Viya Monitoring for Kubernetes deploys Grafana, it is already configured to use Prometheus as a data source for metrics, so these alerts appear automatically without any additional config.   Click on the "Contact points" tab. This is where we configure our channels. A default one, grafana-default-email, is pre-defined for us. Click the pencil icon to open the edit page, where you can select the contact point type (Email) and then specify the destination address. Click the Test button after doing so to make sure everything is set up correctly for the recipient to receive the notification.     Save the changes.   Now head back to the Alert rules tab. The new alert we create in this page will be a Grafana managed alert (i.e. not a PrometheusRule). Click on "+ New alert rule" under Grafana alerts to begin adding the alert. Define the alert conditions based on SAS Viya metrics collected by Prometheus (using either the builder or code methods). Grafana's interface for creating alerts provides a rich set of options for controlling all aspects of your alert condition, and includes the best features of the Prometheus Expression Browser and then some. Specify or select the appropriate metrics, choose how they are filtered and evaluated, and then set the threshold values and evaluation frequency. For example, you could set a rule to trigger when memory usage exceeds a certain value for a specified duration.   Craft an informative rule name and message as well as other descriptive attributes that clearly indicate the issue. You will also be required to select (or create) a folder and group for your rule. In the "Notifications" section, select the key-value labels that will connect the alert to contact points (the equivalent of Alertmanager's routing function).   Save the alert. When the evaluation period lapses, and the alert condition is met, the alert will begin firing and a notification will be sent.         Adjust the threshold values, duration, or other settings as needed based on your alerting strategy and requirements.   We'll look at next steps and additional options in a future post. Stay tuned.   Thank you for reading.   ... View more

As part of ongoing efforts to enhance SAS Viya administration capabilities, we will introduce in this blog post a new program within the pyviyatools repository. getactivityrecords.py is a Python script designed to retrieve and analyze activity records from the SAS Viya environment. These activity records are a kind of audit record that provide insights into the operations and usage patterns of your SAS Viya deployment, enabling administrators and audit compliance folks to better understand at a high-level how the Viya deployment is being used by end users.   The concept of activities has been introduced within SAS Viya's audit service to provide a way to inject higher-level business logic into the auditing process. Traditionally, auditing focuses on low-level technical details, providing limited visibility into the actual usage patterns of different endpoints, making it somewhat challenging to derive meaningful insights. With the introduction of activities, the auditing framework gains a new, higher-level perspective to make auditing more intuitive and understandable. This improved understanding allows for better analysis, troubleshooting, and decision-making based on the audit data.   The audit service routinely loads activity data, which is stored in PostgreSQL, into the SystemData.AUDIT_ACTIVITIES table in CAS for reporting. The User Activity VA report provides a way for users to visualise audit data, but there is not yet a way to do the same for activity data, nor to view the raw activity records themselves.   Enter getactivityrecords.py, the latest addition to the growing pyviyatools repository.   To get started with getactivityrecords.py, you can simply clone the pyviyatools GitHub repository and install the necessary dependencies. When it is run, getactivityrecords.py leverages the SAS Viya REST API to retrieve activity records from the SAS Infrastructure Data server. It also allows you to specify the time range and filters to narrow down the scope of the records you want to retrieve. These records can not yet be surfaced from the sas-viya CLI's audit plug-in, which instead retrieves only the 'regular' audit records.   The program supports multiple output formats, including CSV (default), JSON, and Pandas DataFrames. This flexibility allows you to seamlessly integrate the retrieved activity records into your preferred reporting tools or visualization platforms. Leveraging scheduling tools like cron, you can automate the execution of getactivityrecords.py at regular intervals, enabling the continuous capture of activities within the SAS Viya environment.   The simplest way to execute the program is to simply run it without any additional parameters: getactivityrecords.py     Output:   id ,type ,action ,administrativeAction ,state ,user ,application ,timeStamp ,remoteAddress "2e88b242-1e6b-4aae-bab0-5cdb740d5aee","resource","create","False","success","dagentsrv-gelcorp","identities","2023-06-13T09:15:34.471Z","10.42.1.13" "8d6a74dc-5219-41b4-aaf7-538b24a8e41a","resource","create","False","success","dagentsrv-gelcorp","identities","2023-06-13T09:15:35.877Z","10.42.1.13" "e40bf1ec-47db-4d95-89a0-aa7d3a5204a4","security","login","False","success","geladm","SASLogon","2023-06-13T09:21:54.678Z","127.0.0.1" "efd0af8e-6439-4a3e-97e9-fe50efa5a163","security","login","False","success","geladm","SASLogon","2023-06-13T09:21:55.61Z","127.0.0.1" "5394eeb4-a321-4a3a-bd2c-0b7a4cb01438","security","login","False","success","geladm","SASLogon","2023-06-13T09:22:02.093Z","127.0.0.1" "37e04c46-693f-4899-bb74-3a84a593f184","security","login","False","success","geladm","SASLogon","2023-06-13T09:22:03.771Z","127.0.0.1" "0d4d1a45-2b12-4399-89ba-57de5a6753c7","security","login","False","success","geladm","SASLogon","2023-06-13T09:22:04.449Z","127.0.0.1" "f2ce150d-613b-48c4-812f-7cb86ecad433","security","login","False","success","geladm","SASLogon","2023-06-13T09:22:05.314Z","127.0.0.1" "de62ccdc-1e42-403c-a047-3676812219cc","security","login","False","success","geladm","SASLogon","2023-06-13T09:22:12.019Z","127.0.0.1" "84f530bf-4f93-4b78-893e-e200bf0cba68","security","login","False","success","geladm","SASLogon","2023-06-19T02:53:41.367Z","127.0.0.1" "be704891-ac56-4646-bd89-18206597a4ab","security","SessionAuthenticationSuccess","False","success","geladm","SASLogon","2023-06-19T02:53:41.375Z","127.0.0.1" "fbad594a-8571-4e32-83b3-95204d90d930","security","AdministrativeAction","True","success","geladm","SASLogon","2023-06-19T02:53:48.372Z","127.0.0.1" "e8556c28-e72d-4f3f-b281-207f12f15f68","security","login","False","success","geladm","SASLogon","2023-06-19T03:22:09.338Z","127.0.0.1" "4591fdf7-25c6-4ea2-b705-ab60e89b5fbf","security","SessionDestroyed","True","success","geladm","SASLogon","2023-06-19T03:24:38.374Z","127.0.0.1" "c8005e77-0250-4287-bd45-5eda35238687","security","login","False","success","geladm","SASLogon","2023-06-19T04:28:44.194Z","127.0.0.1" "7a012678-70ae-424c-9088-c5890e9a2d7c","security","SessionAuthenticationSuccess","False","success","geladm","SASLogon","2023-06-19T04:28:44.198Z","127.0.0.1" "d85ebff4-74e4-4762-8b3c-3802a43a2504","security","AdministrativeAction","True","success","geladm","SASLogon","2023-06-19T04:28:45.906Z","127.0.0.1" "aa407dcb-50fb-4951-86ca-5e39c9db5989","security","login","False","success","Ahmed","SASLogon","2023-06-19T04:31:02.689Z","127.0.0.1" "15a68ff7-a6cc-4b18-a8df-be770269cf61","security","SessionAuthenticationSuccess","False","success","Ahmed","SASLogon","2023-06-19T04:31:02.691Z","127.0.0.1" "4c8f6b3b-32ca-4f00-b3fe-4d4a482624ad","security","SessionDestroyed","False","success","Ahmed","SASLogon","2023-06-19T04:37:00.375Z","127.0.0.1" "cf42e5fa-ea42-430d-b8a7-c095be8c9194","security","login","False","success","Delilah","SASLogon","2023-06-19T04:37:05.142Z","127.0.0.1" "17404791-57b0-4f62-9c1d-5c763ec4ceda","security","SessionAuthenticationSuccess","False","success","Delilah","SASLogon","2023-06-19T04:37:05.149Z","127.0.0.1" "50d7eb6e-89d9-42f3-817a-591dde1ca3af","security","SessionDestroyed","True","success","geladm","SASLogon","2023-06-19T04:59:38.474Z","127.0.0.1" "6a44bbf8-b0a3-4171-9e02-337364bdeb6e","security","login","False","success","Delilah","SASLogon","2023-06-19T05:22:14.95Z","unknown" "1af2d753-5235-4cdf-b754-bc5397d2661e","security","SessionDestroyed","False","success","Delilah","SASLogon","2023-06-19T05:32:38.52Z","127.0.0.1"   As the sample output above shows, activity records are currently capturing predominantly security-type records. This is expected to grow in future releases to include records from more and more applications for capturing additional types of user actions.   You can also add additional flags for filtering the output. For example, you can narrow the results by user, application, action, or time range. Consult the built-in documentation to view all options: getactivityrecords.py -h   Output:   usage: getactivityrecords.py [-h] [-l LIMIT] [-t TYPE] [-a APPLICATION] [-c ACTION] [-d ADMIN_ACTION] [-s STATE] [-u USER] [-A AFTER] [-B BEFORE] [-S SORTBY] [-o {csv,json,simple,simplejson}] optional arguments: -h, --help show this help message and exit -l LIMIT, --limit LIMIT Maximum number of records to display -t TYPE, --type TYPE Filter by entry Type -a APPLICATION, --application APPLICATION Filter by entry Application -c ACTION, --action ACTION Filter by entry Action -d ADMIN_ACTION, --admin-action ADMIN_ACTION Filter by Administrative Action -s STATE, --state STATE Filter by entry State -u USER, --user USER Filter by Username -A AFTER, --after AFTER Filter entries that are created after the specified timestamp. For example: 2020-01-03 or 2020-01-03T18:15Z -B BEFORE, --before BEFORE Filter entries that are created before the specified timestamp. For example: 2020-01-03 or 2020-01-03T18:15Z -S SORTBY, --sortby SORTBY Sort the output ascending by this field -o {csv,json,simple,simplejson}, --output {csv,json,simple,simplejson} Output Style   Understanding how your SAS Viya environment is being used can help SAS Viya administrators to meet regulatory obligations, monitor and review security aspects, troubleshoot effectively and better align platform usage with broader business objectives. By leveraging the power of Python and the flexibility of SAS Viya's REST API, getactivityrecords.py enables the retrieval and analysis of activity records to provide valuable insights into user patterns for administrators and auditors.     Find more articles from SAS Global Enablement and Learning here. ... View more

In today's fast-paced world of infrastructure management, monitoring and alerting systems play a vital role in ensuring the health and performance of applications and services. An earlier post introduced the alerting capabilities provided by Prometheus for SAS Viya, which help administrators identify anomalies and respond to incidents proactively. This refresher article provides a high-level recap of what is available out of the box, what it can help with, and covers the process of creating new, custom alerts to enhance real-time monitoring.   Alerting is the process of triggering actions when defined conditions are met. In a SAS Viya deployment (with SAS Viya Monitoring for Kubernetes deployed), those conditions are based on metric information collected by Prometheus (about health and performance of your SAS Viya deployment) or log data collected by OpenSearch.   Alerts can help administrators proactively detect and predict potential anomalies with the deployment and/or infrastructure. When a defined alert condition or threshold is met, Prometheus lets the companion Alertmanager module know. Alertmanager can send notifications to a variety of channels, as well as take other actions automatically when an alert is triggered (such as creating a JIRA or ServiceNow ticket, or even autoscaling up/down). It also provides a facility for managing the lifecycle of triggered alerts (for example, to supress alerts while they’re being investigated).   So what kind of conditions can trigger alerts? When Prometheus is deployed, it includes a heap of generally useful alerts for common conditions out-of-the-box. These include alerts for scenarios like pods stuck in a CrashLoopBackOff state, StatefulSets not matching the expected number of replicas, Persistent Volumes filling up, or elevated CPU throttling. But administrators can also supplement these with their own bespoke alerts.   Before we get to the "how", let’s look at the type of metric information available in Prometheus. This can be done by submitting PromQL (the native, SQL-based querying language) expressions in the Prometheus Expression Browser web application. Conveniently, the Expression Browser includes a metrics explorer and has auto-complete, so you can begin by just selecting a metric and viewing the data available, and then adjusting your expression to filter for what you’re interested in seeing (and alerting on).   For example, assume we want to see memory usage for the sas-cas-control pod. If the word “memory” is typed into the query box, matching metrics are displayed and can be selected from the drop-down box. The container_memory_usage_bytes metric looks like a good match for our use case. A filter can be added by appending a curly brace to the query. We want to filter by pod, so when we start typing pod=" , auto-complete again lets us click on the desired pod name from a drop-down list.   In this example, our resulting expression is container_memory_usage_bytes{pod="sas-cas-control-6c9455dd78-g58fr"} . If we hit the Execute button, we get the metric data values (in bytes) for the query.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   Assume we want to create an alert for this metric. For instance, say we want an alert to be triggered when the sas-cas-control pod exceeds 80% of its memory limit, which we can find by running a kubectl describe on the pod (or for bonus points, try using the Expression Browser to find the appropriate metric and filters that will also give you the limit). Our expression would then become something like [memory usage] / [memory limit] as a percentage, which translates to PromQL as: (sum by (pod) (container_memory_usage_bytes{pod="sas-cas-control-6c9455dd78-g58fr"})) / (sum by (pod) (kube_pod_container_resource_limits{pod="sas-cas-control-6c9455dd78-g58fr",resource="memory"})) * 100   We can then add an operator to our expression to define the alert threshold - if we want to alert the trigger when the usage reaches 80% of the limit, the expression becomes: (sum by (pod) (container_memory_usage_bytes{pod="sas-cas-control-6c9455dd78-g58fr"})) / (sum by (pod) (kube_pod_container_resource_limits{pod="sas-cas-control-6c9455dd78-g58fr",resource="memory"})) * 100 > 80   The process for creating the alert is outlined in an earlier post, but to recap, requires a new YAML file which defines the alert in a PrometheusRule (custom resource) definition to be created and submitted with kubectl:     tee ~/PrometheusRule.yaml > /dev/null << EOF apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: labels: prometheus: prometheus-v4m-prometheus-0 role: alert-rules name: prometheus-viya-rules namespace: monitoring spec: groups: - name: custom-viya-alerts rules: - alert: CASControlMemoryApproachingLimit annotations: description: sas-cas-control container memory usage is approaching its limit. This can result in OOM errors and CAS sessions closing unexpectedly. summary: CAS Control high memory usage expr: (sum by (pod) (container_memory_usage_bytes{pod="sas-cas-control-6c9455dd78-g58fr"})) / (sum by (pod) (kube_pod_container_resource_limits{pod="sas-cas-control-6c9455dd78-g58fr",resource="memory"})) * 100 > 80 labels: severity: critical EOF kubectl -n monitoring create --filename ./PrometheusRule.yaml       Once created, the alert will be visible in the Prometheus UI's Alerts page:     If it when it triggers, it will turn red, and will also appear in the separate Alertmanager UI. When alert conditions are addressed/remediated, the alerts automatically return to a state of “Inactive” (not firing) without any intervention from an administrator. Administrators can "silence" firing alerts in the Alertmanager UI as a way of acknowledging them while they are being investigated.   Obviously, admins still need to specify what happens when the alert does trigger – for example, deciding who gets notified and how. This requires modifying the Alertmanager configuration. A good starting point is this post on alert routing. There are several earlier articles which outline how to configure Alertmanager to integrate with various third-party applications such as MS Teams, and Slack. . The command-line based amtool can also be very useful here.   Leveraging the powerful and flexible alerting capabilities provided by Prometheus in SAS Viya can enable organizations to stay ahead of critical situations, respond promptly to incidents, and optimize their operational efficiency.   For additional information, refer to the official Prometheus documentation and the SAS documentation on SAS Viya’s logging and monitoring. ... View more

One of the key features of SAS Viya is its ability to integrate with different programming languages, including Python. SAS users can submit Python code for execution in SAS Viya with the PYTHON procedure; check out this blog post for an overview of what's possible. Some configuration changes are required to get set up, after which users can start running Python code from a SAS Compute session. That's great for submitting from SAS Studio, but what about in batch?   The necessary setup is covered nicely by Scott McCauley in his article. When the steps are completed, users can submit Python code or programs for execution in a SAS Compute pod (e.g. in a SAS Studio session).   However, a peek inside the file site-config/sas-open-source-config/python/python-transformer.yaml suggests that these configuration changes will only apply to the "sas-compute-job-config" PodTemplate; that's the Custom Resources that tells Kubernetes how to initialise a SAS Compute pod. Any new SAS Compute pod that gets launched will contain these changes to support Python code execution. However, things work a little differently with batch submission of code. The sas-viya CLI has a batch plugin that allows SAS programs (which can contain a PROC PYTHON statement to embed Python code) to be submitted from the command line. With this method, the submitting user must specify a batch context (as opposed to a compute context) to use. By default, Viya is deployed with two batch contexts; default and default-cmd. Both of these use the "sas-batch-pod-template" PodTemplate to initialise a batch session to execute the request. Unfortunately, this template will not contain the changes made to the compute template that are necessary to run Python or R.   We can see this difference by comparing the the compute PodTemplate with the batch PodTemplate.   Running kubectl describe podtemplate sas-compute-job-config shows all the Python configuration, such as the required volume mounts, is present:   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   Those are missing in the output shown by running kubectl describe podtemplate sas-batch-pod-template . This means that when we try to run a SAS program containing Python code, it will fail, since none of the required configuration will exist in the pod launched to run the code.   The solution, then, is to modify the batch PodTemplate so that batch sessions also contain the required open source configuration.  We can just copy and paste the lines that already exist in the transformer for patching the compute PodTemplate to also patch the batch PodTemplate.   The lines of interest are towards the end of python-transformer.yaml:     apiVersion: builtin kind: PatchTransformer metadata: name: compute-job-python-transformer patch: |- # Add python volume - op: add path: /template/spec/volumes/- value: name: python-volume persistentVolumeClaim: claimName: sas-pyconfig # Add mount path for python - op: add path: /template/spec/containers/0/volumeMounts/- value: name: python-volume mountPath: /opt/sas/viya/home/sas-pyconfig readOnly: true # Add python-config configMap - op: add path: /template/spec/containers/0/envFrom/- value: configMapRef: name: sas-open-source-config-python target: kind: PodTemplate name: sas-compute-job-config version: v1     After copy and pasting the above section, and then editing to target the batch PodTemplate, the end of your python-transformer.yaml should now also include:     ... --- apiVersion: builtin kind: PatchTransformer metadata: name: enable-python-batch-transformer patch: |- # Add python volume - op: add path: /template/spec/volumes/- value: name: python-volume persistentVolumeClaim: claimName: sas-pyconfig # Add mount path for python - op: add path: /template/spec/containers/0/volumeMounts/- value: name: python-volume mountPath: /opt/sas/viya/home/sas-pyconfig readOnly: true # Add python-config configMap - op: add path: /template/spec/containers/0/envFrom/- value: configMapRef: name: sas-open-source-config-python target: kind: PodTemplate name: sas-batch-pod-template version: v1     If you've done all the earlier pre-req steps, your kustomization.yaml file won't require any further changes, so you can just build and apply (or deploy your sasdeployment custom resource if you have the SAS Deployment Operator configured).   Now let's verify the changes. When deployment completes, run kubectl describe podtemplate sas-batch-pod-template|grep -A5 -B5 sas-pyconfig to check that the batch PodTemplate has been updated:     k:{"mountPath":"/opt/sas/viya/home/commonfiles"}: .: f:mountPath: f:name: f:readOnly: k:{"mountPath":"/opt/sas/viya/home/sas-pyconfig"}: .: f:mountPath: f:name: f:readOnly: k:{"mountPath":"/sashelp"}: -- Mount Path: /opt/sas/viya/config/etc/SASSecurityCertificateFramework/private Name: security Sub Path: private Mount Path: /gelcontent Name: sas-viya-gelcorp-volume Mount Path: /opt/sas/viya/home/sas-pyconfig Name: python-volume Read Only: true Dns Policy: ClusterFirst Image Pull Secrets: Name: sas-image-pull-secrets-k95ckd46h8 -- Nfs: Path: /shared/gelcontent Server: pdcesx02072.race.sas.com Name: python-volume Persistent Volume Claim: Claim Name: sas-pyconfig Events:     Much better. Then the final test; checking to see if batch submission works.   The program we want to run looks like:     proc python; submit; print('hello world') endsubmit; run;     The command that runs it looks like: sas-viya batch jobs submit-pgm --pgm-path /shared/gelcontent/gelcorp/shared/code/my_code.sas --context default --watch-output --wait-log-list --results-dir ~/   And that results in:     >>> The file set "JOB_20230406_102344_048_1" was created. >>> Uploading "test.sas". >>> The job "1dda5231-1978-4ab0-9da4-d0e264fdecde" was submitted. >>> Waiting for the job to complete... >>> Job started. 1 The SAS System Thursday, April 6, 2023 10:23:00 AM NOTE: Copyright (c) 2016 by SAS Institute Inc., Cary, NC, USA. NOTE: SAS (r) Proprietary Software V.04.00 (TS M0 MBCS3170) Licensed to THIS ORDER IS FOR SAS INTERNAL USE ONLY (SIMPLE), Site 00000000. NOTE: This session is executing on the Linux 3.10.0-1062.12.1.el7.x86_64 (LIN X64) platform. NOTE: Additional host information: Linux LIN X64 3.10.0-1062.12.1.el7.x86_64 #1 SMP Tue Feb 4 23:02:59 UTC 2020 x86_64 Red Hat Enterprise Linux release 8.7 (Ootpa) NOTE: SAS initialization used: real time 0.02 seconds cpu time 0.02 seconds NOTE: AUTOEXEC processing beginning; file is /opt/sas/viya/config/etc/sasgrid/default/autoexec.sas. NOTE: DATA statement used (Total process time): real time 0.00 seconds cpu time 0.00 seconds NOTE: AUTOEXEC processing completed. NOTE: The LOCKDOWN option has been set. SAS is now in the lockdown state. NOTE: The LOCKDOWN option has been set. SAS is now in the lockdown state. 1 proc python; 2 submit NOTE: Python initialized. Python 3.8.13 (default, Apr 4 2023, 04:53:36) [GCC 8.5.0 20210514 (Red Hat 8.5.0-16)] on linux Type "help", "copyright", "credits" or "license" for more information. >>> >>> 2 ! ; 3 print('hello world') 4 endsubmit; 5 run; 6 %let SAS_workpath = %sysfunc(pathname(work)); >>> hello world >>> NOTE: PROCEDURE PYTHON used (Total process time): real time 1.36 seconds cpu time 0.01 seconds NOTE: SAS Institute Inc., SAS Campus Drive, Cary, NC USA 27513-2414 NOTE: The SAS System used: real time 1.40 seconds cpu time 0.05 seconds 2 The SAS System Thursday, April 6, 2023 10:23:00 AM <<< Job ended. <<< Downloading the files in the file set to the following path: "/home/cloud-user//JOB_20230406_102344_048_1". <<< Downloading "test.log". <<< Downloading "test.sas". <<< Copying the log and list files from "/home/cloud-user//JOB_20230406_102344_048_1" to the current directory. <<< Copying "test.log". <<< Deleting the results directory. <<< Deleting the batch job. <<< Deleting the file set.     Success!   There's probably a solid enough case for having the open source configuration applied to both the compute and batch PodTemplates out-of-the-box with the python-transformer.yaml; which will obviously simplify and reduce the setup tasks required. I will update this post with the revised steps if the default setup changes.    Thanks for reading. My thanks also to my colleaque @Bogdan_Teleuca for his contributions.   Find more articles from SAS Global Enablement and Learning here. ... View more

Effective alerting requires that the right people are notified of issues at the right time and via the most appropriate channels. SAS Viya Administrators and DevOps engineers using Alertmanager to manage alerts in the SAS Viya platform can integrate Alertmanager with several third party party applications to distribute alert notifications. While native support is built-in for some applications, many others are capable of receiving information via webhooks (e.g. alert notifications sent by Alertmanager). In this post, we'll look at how Alertmanager can integrate with Microsoft Teams via webhook, allowing alert notifications to be automatically posted to a Microsoft Teams channel when alerts are triggered.   There are several ways to set up the integration. In any case, as there is no native support for Teams integration in Alertmanager, and because Teams requires messages to be in a certain format (one that Alertmanager doesn't produce its alerts in), a third-party tool must be deployed to act as an intermediary. There are many to choose from, but in this case I used an open-source utility called prometheus-msteams. It is essentially a lightweight web server which receives alert messages from Alertmanager and sends them to a Teams channel in the required format.   But first, the Teams channel must be configured. Generate the unique incoming webhook URL (which is where prometheus-msteams will send messages) in the desired Teams channel by right-clicking on the channel name and clicking Connectors.     Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.    If it's not already added, click Add on the Incoming Webhook Connector. When it's added, click Configure.        Provide a name and then click create to generate the unique URL. Copy/take note of this, as it is required later.         Next, set up prometheus-msteams. There are several ways to deploy it, including as a binary or as a docker container. In my lab, I deployed it to my Kubernetes cluster (in the monitoring namespace deployed by SAS Viya Monitoring for Kubernetes) using the Helm chart. First, I added the chart repo as per the instructions with:   helm repo add prometheus-msteams https://prometheus-msteams.github.io/prometheus-msteams/   Then prepare the configuration file before deployment. This is where webhook URL from Teams must be specified.   tee ~/config.yaml > /dev/null << EOF replicaCount: 1 image: repository: quay.io/prometheusmsteams/prometheus-msteams tag: v1.5.1 connectors: # in alertmanager, this will be used as http://prometheus-msteams:2000/alertmanager – alertmanager: https://sasoffice365.webhook.office.com/webhookb2/e501090a-af3d-426c-bf35-2374d5e7d211@b1c14d5c-3625-45b3-a430-9552373a0c2f/IncomingWebhook/4b5a85a99819453aae87cbce67ea6469/07d2cf3f-f87b-4c4c-a3d6-2f10f2993322 # # Enable metrics for prometheus operator metrics: serviceMonitor: enabled: true additionalLabels: release: prometheus # change this accordingly scrapeInterval: 30s EOF   To deploy, run:   helm upgrade --install prometheus-msteams --namespace v4mmon -f config.yaml prometheus-msteams/prometheus-msteams   That results in:    Release "prometheus-msteams" does not exist. Installing it now. NAME: prometheus-msteams LAST DEPLOYED: Wed Mar 15 06:52:49 2023 NAMESPACE: v4mmon STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: ** Please be patient while the chart is being deployed ** To monitor the deployment, execute the following command: kubectl get pods -l app=prometheus-msteams --namespace v4mmon -w   Check that it has been deployed and is running:   kubectl get po -n v4mmon|grep msteams   The result:   prometheus-msteams-588777df8f-btmq2 1/1 Running 0 92s   The next step is to define a new receiver in the Alertmanager configuration for prometheus-msteams. In my lab, I made it the default receiver for all routes, so all alert notifications will get sent to Teams.   tee ~/alertmanager.yaml > /dev/null << EOF global: smtp_smarthost: $(hostname -f):1025 smtp_from: 'alertmanager@gelcorp.com' smtp_require_tls: false resolve_timeout: 5m route: receiver: prometheus-msteams group_wait: 30s group_interval: 5m repeat_interval: 12h receivers: - name: prometheus-msteams webhook_configs: - url: "http://10.96.3.151:1975/alertmanager" send_resolved: true EOF   Note that in this example, the URL is referring to the IP & port of the node where the prometheus-msteams pod is running. In my lab, I set up the prometheus-msteams service with a NodePort so that I could simulate test alerts using cURL (from outside the cluster).   Apply the updated configuration:   cat <<EOF | kubectl apply -f - apiVersion: v1 data: alertmanager.yaml: $(cat ~/alertmanager.yaml | base64 -w0) kind: Secret metadata: name: alertmanager-v4m-alertmanager namespace: v4mmon type: Opaque EOF   Once the Alertmanager config is applied, it may take a minute to take effect (hint: keep an eye on the Status page in the Alertmanager UI). When it does, notifications will start appearing as posts in the Teams channel when the alerts begin firing.        If you want to customize the way messages are posted, you can do that too by following the documented instructions and examples.   For more information refer to the GitHub page. Alternative approaches are also widely documented online.   If you'd rather use Slack than Teams, check out this previous post. Thanks for reading.   Find more articles from SAS Global Enablement and Learning here. ... View more

promtool is a command-line based utility delivered with Prometheus that performs some functions SAS and Kubernetes administrators will find useful. An earlier post demonstrated amtool, the equivalent CLI tool for Alertmanager. Where amtool focuses on verifying the Alertmanager configuration (including validation of alert notifications), promtool is geared towards querying and validating metrics and rules. When creating custom alerts for your Viya deployment, it can be particularly useful for testing the alert rules before you deploy them.   promtool is included as part of the Prometheus deployment, but can also be installed separately (requires golang) and run from anywhere. When SAS Viya Monitoring for Kubernetes is deployed, promtool can be found inside the Prometheus pod in the monitoring namespace at /bin/promtool.   Let's look at some examples of promtool in action.   Query metrics instantly from the command line   Run an instant PromQL query against your metric data, as you would in the Prometheus Expression Browser UI. The below example queries memory usage for the sas-visual-analytics container and displays the raw time-series metric data.   promtool query instant http://prometheus.gelcorp.com 'container_memory_usage_bytes{container="sas-visual-analytics"} / (1024 * 1024 * 1024)'   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   Query metric data over a range   Similar to above, but with values displayed over a specified period. For example, view the same data but in 30 second intervals over the last 5 minutes:   promtool query range http://prometheus.gelcorp.com 'container_memory_usage_bytes{container="sas-visual-analytics"} / (1024 * 1024 * 1024)' --step=30s --start=$(date -d '5 minutes ago' +'%s') --end=$(date -d 'now' +'%s')     This can be useful for determining where your alert thresholds should be.   Check alert and recording rules   Verify that your Prometheus rules are well-defined before deploying them. Put the rules in a rules file, and then use promtool to check them for syntax errors and misconfigured arguments. This also provides a nice way of validating automatic changes in a CI/CD environment. In this example, rules.yaml looks like:   groups: - name: custom-viya-alerts rules: - alert: ViyaMemoryUsage annotations: description: Total Viya 4 namespace memory usage is more than 20% of total memory capacity. summary: Alerting on Viya 4 namespace Memory usage runbook: https://gelgitlab.race.sas.com/GEL/workshops/PSGEL260-sas-viya-4.0.1-administration/-/blob/master/04_Observability/img/alerting/runbook.md expr: ((sum by (node) (container_memory_usage_bytes{container!~"POD",namespace="gelcorp",pod=~"sas-.+"})) / (sum by (node) (kube_node_status_capacity{resource="memory"})) * 100) > 20 labels: severity: critical - alert: LowAvailability expr: (avg_over_time(up{pod=~"sas-.+"}[1w]) * 100) < 70   The command to validate it:   promtool check rules ./rules.yaml       The result will either shows the number of rules detected or display any errors encountered with syntax or structure. Query labels   Find available values for a specified label name.   promtool query labels http://prometheus.gelcorp.com 'severity'         Again, this one can also be helpful when creating alerts; specifically, when assigning labels to alerts for use when sending alert notifications. Unit testing of alerts   Another very nice feature provided by promtool is the ability to unit test rules. You can simulate conditions to verify that the (new) alerts would be triggered once deployed (e.g. conditions that have never happened before). The detailed steps are beyond the scope of this post (there are many examples available online, including in the offical doc), but the process requires you to create a test file and then pass it in as an argument to the 'test rules' subcommand.   And more...   There are several other functions available with promtool, including a command to verify the Prometheus configuration (useful if making changes). For a full list of options and examples, consult in the built-in documentation or the official Prometheus documentation.   Thanks for taking the time to read.    Find more articles from SAS Global Enablement and Learning here. ... View more

The SAS Deployment Operator makes light work of many deployment and administration tasks. My colleague Mike Goddard wrote a great introductory post about what it is, how it works, and deployment options. In this post, we'll look at the steps required to update the SAS Deployment Operator to keep it running smoothly with all the latest fixes and features.   Updating the updater?   Using the Deployment Operator is the recommended approach for applying updates (patches or version updates) to your SAS Viya deployment. In fact, if you don't have it deployed in your Viya environment, the recommended approach for applying Viya software updates is to first deploy it, and then use it to apply the updates. But what about updating the Deployment Operator itself? The Deployment Operator and the rest of your deployed Viya software can generally be updated independently from each other, and the Deployment Operator is backwards compatible with older versions of Viya. As a general recommendation, consider updating the Deployment Operator at least each time you update your Viya software to the latest version. Deploying the latest and greatest will help keep your environment well-oiled. In some cases, it is actually a requirement to upgrade the Deployment Operator as part of a version update to SAS Viya. The good news is that it is a relatively a straight-forward process!   Step 1: Back up and obtain new Deployment Assets   OK, so technically this is more than one step. The intial step requires you to get new Deployment Assets from the MySAS portal or using the viya4-orders CLI to replace the existing assets. The new assets contain the new transformers and manifests for the operator deployment.   First, back up the existing $operator-deploy directory (in this example, the operator is deployed in cluster-wide mode😞   cp -r /home/cloud-user/project/deploy/operator-deploy /home/cloud-user/project/deploy/operator-deploy-orig   Then delete the operator's existing/old sas-bases directory:   cd /home/cloud-user/project/deploy/operator-deploy rm -rf sas-bases   After getting the new assets (for the later/latest version), extract the tarball to create a new sas-bases directory.   tar xvfz SASViyaV4_9CKY1V_stable_2022.09_20221010.1665437790467_deploymentAssets_2022-10-10T224758.tgz Step 2: Copy new operator files   Copy the required operator files to the top-level $operator-deploy directory and make them writable.   cd /home/cloud-user/project/deploy/operator-deploy sudo cp -r sas-bases/examples/deployment-operator/deploy/* . chmod +w site-config/transformer.yaml   Step 3: Restore YAMLs from backups   Rather than making required updates to the kustomization.yaml and transformer.yaml files manually, restore the files from the backups taken earlier, which already contains the necessary customisations such as setting required ClusterRoleBindings, configuring a mirror registry and choosing between cluster-wide mode and namespace mode.   # restore from backups cp ../operator-deploy-orig/site-config/transformer.yaml site-config/ cp ../operator-deploy-orig/kustomization.yaml .   If necessary, make any additional configuration changes after restoring. Be sure to glance over the deployment instructions to verify whether any new customisations are requried. If you repeat this process the next time you update the operator, new additions will be retained next time around.   Step 4: Deploy   And the last step is to deploy. In cluster-wide mode, specify the name of the operator's namespace.   kustomize build . | kubectl -n sasoperator apply -f -   As demonstrated, the process is essentially a redeployment using new assets (but note that fields in transformers.yaml refer to existing values). No clean up is necessary before or after the update, and the operator should not be stopped prior to commencing the update. The new Deployment Operator will be available once the operator pod starts after you deploy (in the last step). Be sure to read the documentation just in case any additional changes are required to transformer.yaml or kustomization.yaml after restoring them.   Thanks for reading.     ... View more

An earlier post outlined the steps involved in setting up archiving for older audit records out of SAS Viya's Audit service. A new feature has since been added to SAS Viya to optionally enable the purging of these archived audit records once they reach a defined retention period to keep the PV storage clean and tidy. This post demonstrates the required steps.   There are two relevant configuration instances to set up the purging of archived audit records. The first, sas.audit.archive, was reviewed in the previous post, but now contains some additional configurable properties.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.     The *.retention.in.archive properties control the period for which audit and activity records are kept in the archive destination before they are purged. Define your retention period in these fields when setting up archiving. If your customer requires records to be kept for longer, adjust the value as necessary. In the example above, archived audit records more than a day old will be cleaned up by the purge process. Also note that another new property is also now available to define the minimum amount of available space required on storage before archiving occurs. Set your threshold value (a percentage of available space) in the storage.local.remainingSpaceThreshold property as required.   Then there's the new sas.audit.purge configuration instance which toggles the purge function on and off, and defines the frequency at which the purge process will run.       To enable purging, ensure that the enabled switch is turned on, and then define the schedule using cron standard syntax. In the same way that audit records are archived to the attached storage volume according to the schedule defined in the scanSchedule property, the purge process will remove record from the defined archive volume path per the schedule.   If backups of archived audit records (or PVCs) are required, be sure to take them between scheduled purges. Admins can also opt to disable the purge facility and manually delete archived records on an ad-hoc basis, but elevated privileges are required.   Thanks for reading. For more on the Audit service, refer to the SAS Viya: Administration guide.   Find more articles from SAS Global Enablement and Learning here. ... View more

Anyone who’s tried modifying the Alertmanager configuration knows how tedious a process it can be. For one, indentation in the configuration.yaml (or alertmanager.yaml) is important, and very little feedback is provided by Kubernetes or Alertmanager if something is invalid or incorrect. As a result, it can be tricky to determine whether the updates have actually taken effect. When the kubectl command to apply the new configuration is run, a message indicates the configuration was updated. But if there are problems with the configuration, such as indentation or syntax issues, they don’t become apparent until the administrator sees that configuration hasn’t actually been updated (by looking the Alertmanager UI, or worse; by realising that alerts aren’t behaving as they should). Even then, it can be difficult to identify exactly why the configuration wasn’t updated. Where is the invalid element or misplaced line break? Introducing amtool, Alertmanager’s command-line companion that provides, among several other neat functions for interacting with alerts, a way for the configuration to be validated to quickly identify issues.   amtool is a CLI tool bundled with Alertmanager. In SAS Viya Monitoring for Kubernetes, it is deployed and accessible from inside the Alertmanager pod with a command like: kubectl -n v4mmon exec -it alertmanager-v4m-alertmanager-0 -- amtool [command]).   This may not be the most convenient way to run it though, as there are some caveats, of which an obvious one is the requirement for direct access to the cluster (and there are others). Alternatively, if you set up Ingress for Alertmanager (or a NodePort for the service), you can install the amtool CLI anywhere, or even run it as a standalone Docker container. In my lab, I cloned the Prometheus repo and compiled the amtool binary per the instructions (requires Golang to be installed).   Back to our primary use case. Assume we want to update the Alertmanger configuration to define a new receiver. First, let's use amtool to query the existing configuration: amtool --alertmanager.url=http://alertmanager.server.gelcorp.com/ config show   global: resolve_timeout: 5m http_config: follow_redirects: true smtp_hello: localhost smtp_require_tls: true pagerduty_url: https://events.pagerduty.com/v2/enqueue opsgenie_api_url: https://api.opsgenie.com/ wechat_api_url: https://qyapi.weixin.qq.com/cgi-bin/ victorops_api_url: https://alert.victorops.com/integrations/generic/20131114/alert/ route: receiver: "null" group_by: - job continue: false routes: - receiver: "null" match: alertname: Watchdog continue: false group_wait: 30s group_interval: 5m repeat_interval: 12h receivers: - name: "null" templates: - /etc/alertmanager/config/*.tmpl   Let's try adding a new route and receiver by loading a new configuration file (alertmanager.yaml), which looks like:   global: smtp_smarthost: mail.gelcorp.com:1025 smtp_from: 'alertmanager@gelcorp.com' smtp_require_tls: false resolve_timeout: 5m route: receiver: sas-email-alert group_wait: 30s group_interval: 5m repeat_interval: 12h receivers: - name: sas-email-alert email_configs: - to: sasadmin@gelcorp.com headers: Subject: 'New Alert - SAS Viya' send_resolved: true require_tls: false   Create a new YAML manifest to update the secret which stores the base64-encoded Alertmanager configuration, and apply it:   tee ~/alertmanager-secret.yaml > /dev/null << EOF apiVersion: v1 data: alertmanager.yaml: $(echo $(cat ~/alertmanager.yaml | base64 -w0)) kind: Secret metadata: name: alertmanager-v4m-alertmanager namespace: v4mmon type: Opaque EOF kubectl apply -f ~/alertmanager-secret.yaml -n v4mmon   In the output, we get the following message:     Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.       But a look at the Alertmanager configuration shows the old configuration, which doesn’t include our updates. Despite the message printed to the terminal, the new configuration was not applied.   Here’s where amtool can help us troubleshoot. If we run the below command, we can figure out what’s broken: amtool --alertmanager.url=http://alertmanager.server.gelcorp.com/ check-config alertmanager.yaml   The culprit: line 13. Sure enough, the email_configs block has an invalid element; the "to" field is incorrectly indented.       After adding a space to correct it, we can try checking it again. This time, it looks more positive:       This is the config file we should apply to ensure our Alertmanager configuration update is successful.   There are many other useful functions provided by amtool, including:   Displaying alert routes: amtool config routes show   Testing alert routes (in this example, to see where an alert with a specified label will be sent to): amtool config routes test severity=warning Viewing alerts: amtool alert as well as a bunch of other cool stuff like managing silences and testing new alerts.   amtool provides a simple, quick and effective method for managing Alertmanager configuration and artifacts. Instructions can be scripted and scheduled, and very little setup is required to get started. The built-in documentation contains a comprehensive list of possible commands available to the administrators (hint: you can also enable bash auto-completion with: amtool --completion-script-bash ).   Thanks for reading.   Ajmal Farzam     Find more articles from SAS Global Enablement and Learning here. ... View more

SAS Viya provides several mechanisms for integrating the Python language with SAS Viya's data and analytics capabilities. One such tool is SASPy, a module that creates a bridge between Python and SAS, allowing Python developers, who may not necessarily be familiar with SAS code, to leverage the power of SAS directly from a Python client. In this post, we'll look at the setup required to configure SASPy to access SAS Viya (deployed on Kubernetes) from a Jupyter notebook.   What is SASPy?   The open-source SASPy Python module converts Python code to SAS code and runs the code in SAS. It provides Python APIs to SAS so that you can start a SAS session and run analytics from Python. You can move data between SAS data sets and Pandas dataframes and exchange values between python variables and SAS macro variables. You can use the module in both interactive line mode and batch Python, as well as in Jupyter Notebooks. The results include ODS output, and can be returned as Panda data frames. Not all Python methods are supported, but you can customize the module to add or modify methods.   Requirements and Setup   You will need: 1. A SAS Viya environment. I'm running the latest Stable cadence of SAS Viya on Kubernetes, but SASPy supports any version from SAS 9.4 onwards. 2. A Python environment. In my lab, I downloaded and installed Anaconda which includes Python and Jupyter notebook. 3. SASPy. Go to the GitHub repository and download and install by following the instructions. I had an older version of Python installed on my client machine, so to make sure the SASPy module was made available to the 'correct' Python (the one that comes with Anaconda), I launched the command line prompt from the Anaconda Navigator interface and installed SASPy from here by running: pip install saspy .     Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.    To configure, specify the connection parameters in the sascfg_personal.py configuration file for SASPy to connect to your Viya deployment per the instructions. In the config file, you can specify the preferred connection method (for SAS Viya, we're limited to HTTP/HTTPS connections) and authentication information. Mine looks something like:   SAS_config_names=['httpsviya'] SAS_config_options = {'lock_down': False, 'verbose' : True, 'prompt' : True } SAS_output_options = {'output' : 'html5'} # not required unless changing any of the default httpsviya = {'ip' : 'gelcorp.sas.com', 'context' : 'Data Mining compute context', 'authkey' : 'HTTP_Dev_Henrik', 'options' : ["fullstimer", "memsize=1G"] }   Note that in this example, we're using an authinfo file, which is placed in the USERHOME directory and the contents of which look like:   HTTP_Dev_Henrik user Henrik password lnxsas   Demonstration   Once everything is configured, we can try it by running some code. Here's my demo Jupyter notebook...       ...and a running commentary of what happened in 4 steps:   (1) In the first step , we're importing the SASPy libraries. Then we establish a connection to SAS Viya by referring to the connection defined in the config file, and a SAS Compute Server (SAS Programming Runtime) session is launched. As per the (configurable) settings in sascfg_personal.py, the Data Mining context is used. (2) Python procedures are then used to query a sample SAS data set in various ways. SASPy converts the Python code to SAS code, which it executes in the Compute Server session. The results of the queries are returned directly as Pandas dataframes. (3) We also then run a SAS code fragment to embed and run actual SAS code. (4) The session stays open until we kill it with the endsas() function.   By the way, if we look in the logs (using Kibana), we can see the SAS code that actually ran in the compute session:       Additional Resources   Refer to the SASPy documentation for more information about installation, configuration and usage.   Of course, SASPy is not the only way to integrate Python and SAS. The Python SWAT package can be used to interact with CAS directly using the CAS REST API, allowing developers to run CAS actions from Python. Other built-in capabilities and optional modules also exist for things like Deep Learning, ESP, for accessing Python from a SAS program, and more.   Quite a lot of material is available for further information, including several blog posts, Youtube videos (including this excellent tutorial), and the official documentation.   And to wrap up, a note on using SASPy with SAS Workload Management. I have previously written about submitting SAS 9.4 Grid Manager jobs from Python. That capability is now available for any SAS Viya deployment that includes a SAS Workload Management license. In Workload Management, any 'compute' (excluding CAS) workload that is submitted as a job to SAS Workload Orchestrator, which uses its built-in smarts to determine where to send it for execution. So in effect, any SAS Compute Server session is grid-enabled, including sessions launched from Python. The workloads are submmitted as jobs to the Workload Orchestrator Manager which starts a launcher pod on an appropriate node in the cluster to execute the code. An admin can then interact with and manage the jobs as though they were any other job, and can also utilize the additional monitoring and administration functions provided by Workload Management.    Thank you for reading.   Find more articles from SAS Global Enablement and Learning here. ... View more

SAS Viya's Audit service tracks system usage through audit records, which are stored in the SAS Infrastructure Data Server. By default, audit records are retained for seven days; older records are archived every day at midnight (schedule is customizable). But the Audit service runs as a pod in a Kubernetes cluster, and is therefore designed to be ephemeral and disposable. So how does archival (which can be a persistent process) of audit records work? In this post, we'll look at the extra setup steps required to configure audit record archiving.   TL;DR   Attach a PVC to the sas-audit pod and update the Audit service configuration to configure archiving behaviour.   The detail   First, create the YAML manifests for a new PVC which will be attached to the sas-audit pod to store archived audit records. The files will be in a new subdirectory under site-config called sas-audit in your $deploy directory.   mkdir ~/project/deploy/gelcorp/site-config/sas-audit   In this example, we'll create a 1Gi PVC.   tee ~/project/deploy/gelcorp/site-config/sas-audit/resources.yaml > /dev/null << EOF apiVersion: v1 kind: PersistentVolumeClaim metadata: name: audit-archive-pvc spec: accessModes: - ReadWriteMany volumeMode: Filesystem resources: requests: storage: 1Gi # Change this to desired capacity. storageClassName: nfs-client # Change this the desired storage class. EOF   We'll then define how and where it's mounted. In this case, at the path " /archive " in the sas-audit pod.   tee ~/project/deploy/gelcorp/site-config/sas-audit/archive-transformer.yaml > /dev/null << EOF apiVersion: builtin kind: PatchTransformer metadata: name: archive-transformer patch: |- - op: add path: /spec/template/spec/volumes/- value: name: audit-archive-volume persistentVolumeClaim: claimName: audit-archive-pvc - op: add path: /spec/template/spec/containers/0/volumeMounts/- value: name: audit-archive-volume mountPath: /archive target: group: apps kind: Deployment name: sas-audit version: v1 EOF   Now we need to backup and rebuild the kustomization.yaml file (using yq in the below example)...   cd ~/project/deploy/gelcorp cp -p kustomization.yaml kustomization.pre-audit-archive.yaml yq4 eval -i '.transformers += ["site-config/sas-audit/archive-transformer.yaml"]' ./kustomization.yaml yq4 eval -i '.resources += ["site-config/sas-audit/resources.yaml"]' ./kustomization.yaml   ...and apply to the cluster. If you have SAS Deployment Operator, back up and regenerate the SASDeployment CR YAML file and then apply it, otherwise just run a kubectl apply command:   cd ~/project/deploy/ cp -p gelcorp-sasdeployment.yaml gelcorp-sasdeployment.pre-audit-archive.yaml docker run --rm \ -v ${PWD}/license:/license \ -v ${PWD}/gelcorp:/gelcorp \ sas-orchestration \ create sas-deployment-cr \ --deployment-data /license/SASViyaV4_9CKY1V_certs.zip \ --license /license/SASViyaV4_9CKY1V_license.jwt \ --user-content /gelcorp \ --cadence-name lts \ --cadence-version 2021.2 \ --cadence-release 20220104.1641281166167 \ > ~/project/deploy/gelcorp-sasdeployment.yaml kubectl apply -f ~/project/deploy/gelcorp-sasdeployment.yaml Wait for reconciliation (about 5-10 mins), verify the PVC has been mounted in the new pod (the old pod gets terminated):   kubectl exec -it sas-audit-848b79fb7d-ssqmb -- ls / archive dev lib lost+found opt run srv usr bin etc lib64 media proc sbin sys var boot home licenses mnt root security tmp   So now we have somewhere to archive to, we need to tell the Audit service to use it.   Go to SAS Environment Manager > Configuration > Audit service and edit (pencil icon) the sas.audit.archive configuration instance. Update the storage.local.destination property with a path (inside the new archive location), and set storageType to "local". Optionally, adjust the cron schedule and retention periods as required.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   Hit Save to apply. When the time specified in your cron expression is reached, the archive process commences and records will start to be written to the PVC (and removed from the Infrastructure Data Server). You can view them directly from the main cluster node:   ls /srv/nfs/kubedata/gelcorp-audit-archive-pvc-pvc-09d41561-b13b-4438-bcc0-378c388e0fbe/archive/audit audit--1645619701154.arc audit--1645619706866.arc audit--1645619711756.arc audit--1645619716885.arc audit--1645619721857.arc audit--1645619726758.arc audit--1645619731454.arc audit--1645619736548.arc audit--1645619741568.arc audit--1645619746173.arc audit--1645619702309.arc audit--1645619707666.arc audit--1645619712760.arc audit--1645619717673.arc audit--1645619722683.arc audit--1645619727558.arc audit--1645619732241.arc audit--1645619737355.arc audit--1645619742358.arc audit--1645619746963.arc audit--1645619703291.arc audit--1645619708476.arc audit--1645619713546.arc audit--1645619718498.arc audit--1645619723663.arc audit--1645619728371.arc audit--1645619732983.arc audit--1645619738084.arc audit--1645619743051.arc audit--1645619747661.arc audit--1645619704236.arc audit--1645619709298.arc audit--1645619714360.arc audit--1645619719357.arc audit--1645619724463.arc audit--1645619729059.arc audit--1645619733762.arc audit--1645619738860.arc audit--1645619743752.arc audit--1645619748484.arc audit--1645619705147.arc audit--1645619710060.arc audit--1645619715156.arc audit--1645619720184.arc audit--1645619725255.arc audit--1645619729865.arc audit--1645619734630.arc audit--1645619739647.arc audit--1645619744567.arc audit--1645619706056.arc audit--1645619710867.arc audit--1645619715946.arc audit--1645619721071.arc audit--1645619725981.arc audit--1645619730639.arc audit--1645619735760.arc audit--1645619740385.arc audit--1645619745361.arc   Each .ARC is a JSON file containing audit data containing 1000 records (as defined by the batchSize configuration property). The files can be copied to another location (e.g. somewhere that gets backed up) and/or deleted, noting that they are owned by root (sudo access required).   Thank you for reading.    Find more articles from SAS Global Enablement and Learning here. ... View more