As part of SAS Viya Monitoring for Kubernetes, Grafana runs alongside Prometheus to keep an eye on a SAS Viya deployment. Grafana includes built‑in support for several alert “receivers” out of the box, so that notifications can be sent to administrators and engineers via email, Slack, Microsoft Teams, and more. It's also possible to use webhooks to configure additional applications and platforms as destinations for alert notifications. In this post, I will cover the steps required to be notified of alerts via WhatsApp, including important considerations and additional setup.   My existing deployment has SAS Viya deployed to a namespace called 'edu', and the monitoring stack of SAS Viya Monitoring for Kubernetes (including Grafana) deployed to a namespace called 'monitoring'. To send alerts to WhatsApp, some more setup is required. First, we need a way to send messages to WhatsApp - that is, a commercial service or platform that can send messages to users. For my demo, I used Twilio, which has a limited free tier, but offers a paid service better suited to enterprises and for Production use. Twilio (and other similar services - there are many) is a WhatsApp Business API provider. By default, however, WhatsApp's API isn't webhook friendly, so we also need something else that can relay the alert payload from Grafana. That's the second required part of the setup - a bridge service that accepts Grafana webhooks and forwards them to Twilio.   After creating an account in Twilio or your WhatsApp Business API provider platform, locate the account information required for the bridge service. In my case, I needed:   the Account SID – a value starting with AC. the Auth Token – shown alongside the Account SID when you reveal it. the WhatsApp sender number. the recipient number(s). I used my own. You'll need to verify recipient numbers.   These values will be injected as environment variables into the bridge service. The bridge is a small HTTP service that accepts alert notifications from Grafana and calls (in my case) Twilio's WhatsApp API. I asked ChatGPT to write the Python code for me and create a docker container:   mkdir ~/bridge cd ~/bridge # Create two files inside this folder: app.py and requirements.txt # ------ app.py ------ cat > app.py << 'EOF' import os import requests from fastapi import FastAPI, Request, HTTPException app = FastAPI() ACCOUNT_SID = os.environ["ACCOUNT_SID"] AUTH_TOKEN = os.environ["AUTH_TOKEN"] FROM_NUMBER = os.environ["FROM_NUMBER"] # e.g. whatsapp:+14100000000 TO_NUMBER = os.environ["TO_NUMBER"] # e.g. whatsapp:+61XXXXXXXXX @app.post("/grafana-alert") async def grafana_alert(req: Request): body = await req.json() # Grafana-style payload: status + alerts[] status = body.get("status", "unknown") alerts = body.get("alerts", []) first = alerts[0] if alerts else {} labels = first.get("labels", {}) annotations = first.get("annotations", {}) title = labels.get("alertname") or annotations.get("summary") or "Grafana alert" severity = labels.get("severity", "n/a") summary = annotations.get("description") or annotations.get("summary", "") text = f"ALERT: {title}\nStatus: {status}\nSeverity: {severity}\n{summary}" # Twilio WhatsApp API url = f"https://api.twilio.com/2010-04-01/Accounts/{ACCOUNT_SID}/Messages.json" data = { "To": TO_NUMBER, "From": FROM_NUMBER, "Body": text, } resp = requests.post(url, data=data, auth=(ACCOUNT_SID, AUTH_TOKEN)) if resp.status_code >= 300: raise HTTPException(status_code=500, detail=f"Twilio error {resp.status_code}: {resp.text}") return {"ok": True} EOF # ------ requirements.txt -------- cat > requirements.txt << 'EOF' fastapi uvicorn[standard] requests EOF   # --- create dockerfile in the same folder cat > Dockerfile << 'EOF' FROM python:3.12-slim WORKDIR /app COPY requirements.txt . RUN pip install --no-cache-dir -r requirements.txt COPY app.py . EXPOSE 8080 CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8080"] EOF   # ----- replace $DOCKER_USER with your Docker Hub or other registry username and then build container docker build -t $DOCKER_USER/grafana-whatsapp-bridge:v1 .   The bridge expects Grafana’s default webhook JSON, extracts a title, status, severity, and summary, (all configurable) builds a message string, and posts it to Twilio’s WhatsApp Messages API. Run it bridge with your Twilio credentials as environment variables:   docker run --rm -p 8080:8080 \ -e ACCOUNT_SID=ACXXXXXXX \ -e AUTH_TOKEN=de0c290000000000 \ -e FROM_NUMBER='whatsapp:+141XXXXXXXX' \ -e TO_NUMBER='whatsapp:+61XXXXXXXXX' \ $DOCKER_USER/grafana-whatsapp-bridge:v1   Test it from a second terminal session:   curl -X POST http://localhost:8080/grafana-alert \ -H 'Content-Type: application/json' \ -d '{ "status": "firing", "alerts": [ { "labels": { "alertname": "TestAlert", "severity": "critical" }, "annotations": { "summary": "Test alert from curl" } } ] }'   If everything is configured correctly, you should receive a WhatsApp message containing the test alert details. At this point, you can adjust things like the message string, included labels, etc. I opted to push the image to my registry so that I can easily deploy it into my Kubernetes cluster to avoid having to troubleshooting networking issues; BUT this is a throwaway demo environment, and you'll need to consider the appropriate place for the bridge to be deployed in your environment and plan/secure accordingly.   docker push $DOCKER_USER/grafana-whatsapp-bridge:v1   Before we deploy the bridge into the cluster, we need to make sure it will be able to safely read the Twilio connection details. We can put all that info in a secret:   cat > twilio-secret.yaml << 'EOF' apiVersion: v1 kind: Secret metadata: name: twilio-whatsapp-secret namespace: monitoring type: Opaque stringData: ACCOUNT_SID: "ACXXXXXXX" AUTH_TOKEN: "de0c290000000000" FROM_NUMBER: "whatsapp:+141XXXXXXXX" TO_NUMBER: "whatsapp:+61XXXXXXXXX" EOF # apply kubectl apply -f twilio-secret.yaml   Now we can deploy the container into the same Kubernetes cluster as SAS Viya and Grafana. In my demo, I kept it simple by deploying it to the monitoring namespace, but that's not recommended for Production environments.   cat > whatsapp-bridge.yaml << 'EOF' apiVersion: apps/v1 kind: Deployment metadata: name: whatsapp-bridge namespace: monitoring spec: replicas: 1 selector: matchLabels: app: whatsapp-bridge template: metadata: labels: app: whatsapp-bridge spec: containers: - name: whatsapp-bridge image: /grafana-whatsapp-bridge:v1 ports: - containerPort: 8080 envFrom: - secretRef: name: twilio-whatsapp-secret --- apiVersion: v1 kind: Service metadata: name: whatsapp-bridge namespace: monitoring spec: selector: app: whatsapp-bridge ports: - port: 8080 targetPort: 8080 EOF # apply kubectl apply -f whatsapp-bridge.yaml kubectl get pods -n monitoring -l app=whatsapp-bridge   When the pod is running, the service is reachable internally at:   http://whatsapp-bridge.monitoring.svc.cluster.local:8080/grafana-alert    Now, with the bridge up and running in the cluster, we can configure Grafana to connect to it via a webhook contact point. I've covered this in a previous post, but the high-level steps are:   In the Grafana UI, go to Alerting -> Contact points. Click Create contact point. Set: Name: whatsapp-bridge Type: Webhook URL: http://whatsapp-bridge.monitoring.svc.cluster.local:8080/grafana-alert  Set the HTTP method to POST and keep the default JSON body. Save the contact point.   You can test the contact point at this point to confirm that Grafana can reach the bridge and that WhatsApp notifications work. There's a 'Test' button on the editor page. If everything is wired correctly, you should see a WhatsApp message appear on your phone a few seconds later.   Now, when alerts are created in Grafana, the contact point can be set to whatsapp-bridge in the 'Configure Notifications' pane:   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   When an alert using that contact point triggers, a WhatsApp message arrives with details we've specified:     From here you can refine the message format, restrict WhatsApp alerts to only critical severities, or extend the bridge to send messages to multiple recipients or additional downstream systems. In real-world environments, some more planning and prep is required with particular consideration given to security, networking and to additional costs that may be incurred with this setup.     Find more articles from SAS Global Enablement and Learning here. ... View more

In tightly controlled dark‑site or air-gapped environments, SAS Viya administrators often need a way to discover software updates and patches without giving the platform unrestricted internet access. The SAS Update Checker can solve this by checking for available updates through a proxy so that all outbound traffic is controlled and auditable.     Why Use a Proxy with Update Checker?   In many organisations, Kubernetes worker nodes are not allowed to talk directly to the internet and must send web traffic via a proxy. Configuring SAS Update Checker to leverage the proxy allows for querying for updates in a safe way:   Enforces allow‑lists of destinations (for example, only cr.sas.com and other SAS endpoints), reducing attack surface. Allows security teams to see which external services are contacted and when. In environments with mirror registries, Update Checker can still query SAS registries via the proxy while images for deployments/updates are pulled from a local mirror.   SAS provides a sample overlay in deployment assets to configure the Update Checker cronjob with some additional parameters to talk to a proxy server.   In my demo environment, I deployed a simple Tinyproxy container inside my Kubernetes cluster and exposed it via a ClusterIP Service, and then configured Update Checker to send all outbound requests through that service. This same pattern can be applied to more complex setups, where Tinyproxy would be replaced by a hardened or corporate proxy, which could be deployed anywhere.   In my case, the proxy service was exposed at http://tinyproxy.proxy-test.svc.cluster.local:3128.   I could then verify connectivity with:   kubectl run nettest --rm -it \ --image=curlimages/curl \ --restart=Never -- sh curl -v -x http://tinyproxy.proxy-test.svc.cluster.local:3128 https://cr.sas.com/   The output shows a CONNECT cr.sas.com:443 followed by a normal TLS handshake and HTTP 301/200 response, suggesting that my proxy is reachable and can access SAS endpoints from inside the cluster.   * Host tinyproxy.proxy-test.svc.cluster.local:3128 was resolved. * IPv6: (none) * IPv4: xx.xx.xx.xxx * Trying xx.xx.xx.xxx:3128... * CONNECT: no ALPN negotiated * allocate connect buffer * Establish HTTP proxy tunnel to cr.sas.com:443 > CONNECT cr.sas.com:443 HTTP/1.1 > Host: cr.sas.com:443 > User-Agent: curl/8.17.0 > Proxy-Connection: Keep-Alive > < HTTP/1.0 200 Connection established < Proxy-agent: tinyproxy/1.11.0 < * CONNECT phase completed * CONNECT tunnel established, response 200 * ALPN: curl offers h2,http/1.1 * TLSv1.3 (OUT), TLS handshake, Client hello (1): * SSL Trust Anchors: * CAfile: /cacert.pem * TLSv1.3 (IN), TLS handshake, Server hello (2): * TLSv1.2 (IN), TLS handshake, Certificate (11): * TLSv1.2 (IN), TLS handshake, Server key exchange (12): * TLSv1.2 (IN), TLS handshake, Server finished (14): * TLSv1.2 (OUT), TLS handshake, Client key exchange (16): * TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1): * TLSv1.2 (OUT), TLS handshake, Finished (20): * TLSv1.2 (IN), TLS handshake, Finished (20): * SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256 / secp256r1 / rsaEncryption ...     Configuring SAS Update Checker to Use the Proxy   SAS Update Checker runs as a Kubernetes CronJob that uses the sas-orchestration image to query SAS services and generate a report of available updates and hot fixes. To route all of its outbound traffic through the proxy, add the required proxy environment variables to the container specification in that CronJob by copying the example manifest to site-config and then setting the proxy values for three variables. Don't forget to add the transformer to kustomization.yaml.   In my environment, the Update Checker's env block looks like this:   env: ... - name: HTTP_PROXY value: "http://tinyproxy.proxy-test.svc.cluster.local:3128" - name: HTTPS_PROXY value: "http://tinyproxy.proxy-test.svc.cluster.local:3128" - name: NO_PROXY value: "localhost,127.0.0.1,172.18.0.0/24,.cluster.local,.svc" ...   With this setup, all SAS Update Checker HTTP and HTTPS calls go through the same controlled proxy path, while internal calls stay inside the cluster and are not unnecessarily proxied.     Verifying That Update Checker Uses the Proxy   Once the proxy configuration is in place, it is important to confirm that Update Checker actually uses it. I verified with two simple checks.   1. Observe proxy logs while Update Checker runs.   My Tinyproxy logs show that a connection is made to the SAS Entitlements Service when the Update Checker job kicks off:   CONNECT Dec 19 05:12:30.574 [1]: Connect (file descriptor 5): 10.42.0.155 CONNECT Dec 19 05:12:30.574 [1]: Request (file descriptor 5): CONNECT ses.sas.download:443 HTTP/1.1 INFO Dec 19 05:12:30.574 [1]: No upstream proxy for ses.sas.download INFO Dec 19 05:12:30.574 [1]: opensock: opening connection to ses.sas.download:443 INFO Dec 19 05:12:30.577 [1]: opensock: getaddrinfo returned for ses.sas.download:443 CONNECT Dec 19 05:12:30.578 [1]: Established connection to host "ses.sas.download" using file descriptor 6. INFO Dec 19 05:12:30.578 [1]: Not sending client headers to remote machine INFO Dec 19 05:12:35.615 [1]: Closed connection between local client (fd:5) and remote client (fd:6)   2. Temporarily stop the proxy and confirm connection failure.   Scaling the proxy Deployment to zero replicas makes the service endpoint unreachable:   kubectl scale deploy/tinyproxy -n proxy-test --replicas=0 kubectl create job --from=cronjob/sas-update-checker \ sas-update-checker-proxy-failtest   With the proxy stopped, the Update Checker job log shows a connection failure:   Found 2 pods, using pod/sas-update-checker-proxy-failtest-2q6rn {"level":"info","version":1,"source":"sas-orchestration","messageKey":"sas-orchestration.command.started","messageParameters":{"name":"report"},"properties":{"logger":"internal/apihelpers","caller":"apihelpers/apihelpers.go:44"},"timeStamp":"2025-12-19T05:13:08.670823+00:00","message":"The report command started"} {"level":"info","version":1,"source":"sas-orchestration","messageKey":"sas-orchestration.current.time","messageParameters":{"now":"2025-12-19T05:13:08Z"},"properties":{"logger":"internal/cmd/report","caller":"report/report.go:59"},"timeStamp":"2025-12-19T05:13:08.671198+00:00","message":"Current time is '2025-12-19T05:13:08Z'"} {"level":"info","version":1,"source":"sas-orchestration","messageKey":"sas-orchestration.command.failure","messageParameters":{"name":"report"},"properties":{"logger":"internal/apihelpers","caller":"apihelpers/apihelpers.go:50"},"timeStamp":"2025-12-19T05:13:09.699384+00:00","message":"The report command failed"} Error loading entitlements file: "https://ses.sas.download/ses/entitlements.json" Caused by: * Failed to get 'https://ses.sas.download/ses/entitlements.json' * Get "https://ses.sas.download/ses/entitlements.json": proxyconnect tcp: dial tcp 10.43.25.144:3128: connect: connection refused   When the Update Checker job fails with connection errors and succeeds again once the proxy is scaled back up, it demonstrates a hard dependency on the proxy path. This kind of failure/restore test is a good fit for CI/CD or operational readiness checks before fully hardening a dark‑site environment.     Summary   This simple demonstration shows how administrators can create a bridge between strict network controls and the operational need to stay informed about available updates and patches.   For further reading, these resources are particularly relevant:   SAS Update Checker Report Create a Mirror Registry at an Air-Gapped Site Navigating the Hall of Mirrors Using the SAS Configurator for Open Source in Dark‑Site Environments     Find more articles from SAS Global Enablement and Learning here. ... View more

SAS Viya Monitoring for Kubernetes now offers a set of pre-configured Grafana alerts that can be deployed to enhance monitoring of your SAS Viya platform. These new alerts streamline alerting setup by providing administrators with the option to include SAS Viya-specific alerts right from the moment of deployment, complementing the existing Kubernetes cluster-level alerts deployed with Prometheus to enhance observability and operational readiness. NOTE: The information provided in this article currently applies to non-OpenShift Container Platform deployments of SAS Viya only. The alerting capabilities provided by SAS Viya Monitoring for Kubernetes will be extended to SAS Viya deployments on OpenShift environments soon. New Pre-Configured Grafana Alerts     Beginning in version 1.2.38 (June 2025), the monitoring deployment ships an optional* suite of SAS Viya-specific Grafana alerts. These alerts proactively monitor critical SAS Viya components and services to promptly flag potential issues such as resource exhaustion, pod restarts, and message queue backlogs. Post-deployment, administrators can configure notifiers (such as email, Slack, or SMS, as outlined in this earlier post) via Grafana’s web interface or add custom alerts by placing YAML configuration files from samples/alerts in $USER_DIR/monitoring/alerting/ before running the deployment script (deploy_monitoring_cluster.sh).   Version 1.2.42 (September 2025) also introduced an automatic SMTP server configuration feature, enabling seamless setup of email notifications by toggling the AUTOGENERATE_SMTP flag and providing SMTP server details via environment variables.   * The Grafana-managed alerts were deployed by default with SAS Viya Monitoring for Kubernetes from versions 1.2.38-1.2.41, but are optional from 1.2.42 and can be deployed from the samples/alerts directory.     Sample Alert List   Alert Name Description Condition Catalog DB Connections High Triggers if active catalog database connections exceed 21. In-use DB connections > 21 Crunchy Backrest Repo Usage High Alerts if PostgreSQL backrest repo storage usage is above 50%. Storage usage > 50% Crunchy PGData Usage High Monitors PostgreSQL data directory usage for WAL log accumulation beyond 50%. PGData filesystem > 50% full NFS Share Usage High Signals when NFS storage used by CAS surpasses 85%. NFS share usage > 85% CAS Restart Detected Detects CAS pod restarts by identifying pods with uptime less than 15 minutes. CAS pod uptime < 15 minutes CAS Memory Usage High Flags high memory consumption by CAS pods. Custom threshold (see YAML) Viya Pod Restart Count High Alerts when pods restart more than 20 times. Restart count > 20 Viya Readiness Probe Failed Indicates if critical SAS readiness probes fail. Pod not Ready for critical period RabbitMQ Ready Queue Backlog Warns of excessive backlog in RabbitMQ message queues, possibly affecting downstream processing. Ready messages > 10,000 RabbitMQ Unacked Queue Backlog Checks for unacknowledged RabbitMQ messages > 5,000, suggesting downstream service problems. Unacked messages > 5,000     Customising the Alerts   The alert manifests are provided in standard Prometheus alerting rule format (using PromQL) and can be customised before deployment to suit different SAS Viya environments. Administrators can:   Adjust thresholds for resource usage or restart counts. Modify pod and namespace selectors in rules to target correct resources and refine scope. Tailor alert messages and severity labels for improved notification handling. Change evaluation intervals or durations for condition triggering. Duplicate and extend rule files for more granular monitoring.   The samples/alerts directory of the viya4-monitoring-kubernetes GitHub repository contains README documentation and example alert files that serve as a guide for customisation and best practices. The sample alerts can be copied to USER_DIR/monitoring/alerting and customised to suit your particular SAS Viya environment prior to deployment.     Complementing Prometheus OOTB Alerts   Prometheus, when deployed with SAS Viya Monitoring for Kubernetes, already ships with a set of out-of-the-box rules defined as PrometheusRule resources, which are entirely distinct from (and complementary to) the new SAS-focused set of Grafana-managed sample alerts. These default PrometheusRules monitor common, general Kubernetes health and performance conditions such as:   High memory pressure or CPU utilisation. Node, pod, or container availability. General resource exhaustion and crash loop detection.   These are broadly useful for any Kubernetes workload (not just SAS) and are managed separately from the SAS-specific Grafana-managed sample alerts. Both types of alerts can be viewed in Grafana, and additional custom alerts can be added to both. See my earlier post for a comparison of the alerting features and functions between Grafana and Prometheus Alertmanager.     Summary   The optional set of pre-configured SAS Viya-specific Grafana alerts provides administrators with a powerful, ready-made alerting framework, enhancing platform monitoring beyond generic Kubernetes signals. These alerts are flexible and extensible to fit diverse operational environments and work side-by-side with Prometheus's general alerts for a comprehensive monitoring solution. The added SMTP configuration and notifier flexibility further simplify integrating alerting workflows into organisational communication channels.   The new alerts empower SAS Viya platform teams to detect and respond to issues faster, as well as maintain stability and performance of SAS Viya deployments from the moment the monitoring tools are deployed.     Find more articles from SAS Global Enablement and Learning here. ... View more

Apache Airflow is a powerful orchestration tool that SAS Viya users can leverage to automate, schedule, and monitor SAS (and other) jobs. When Airflow runs in Kubernetes, accessing task execution logs from outside the Airflow UI can be useful for effective monitoring, but finding them can be confusing. This post provides some guidance on options for accessing Airflow logs externally.     Where do Airflow logs live?   This depends on your individual setup and can vary. In my lab environment, Airflow is deployed to a separate namespace in my Kubernetes cluster where SAS Viya also resides. Here, several components are deployed: root@server:~# kubectl get po -n airflow   NAME READY STATUS RESTARTS AGE airflow-api-server-84c4446994-sp8kg 1/1 Running 45 (26h ago) 111d airflow-dag-processor-6b55f97b96-6w64q 2/2 Running 25 (26h ago) 111d airflow-postgresql-0 1/1 Running 21 (26h ago) 111d airflow-redis-0 1/1 Running 4 (26h ago) 111d airflow-scheduler-7d6c44b4d7-9f2mn 2/2 Running 42 (26h ago) 111d airflow-statsd-78b94c6899-r6xlt 1/1 Running 4 (26h ago) 111d airflow-triggerer-0 2/2 Running 24 (26h ago) 111d airflow-worker-0 2/2 Running 14 (26h ago) 111d   In this setup, Airflow tasks are executed inside existing worker pods (airflow-worker-0), which are long-running and already up in the airflow namespace. The workers run multiple tasks inside the same pod/container (Kubernetes does not create new pods per task in my setup).   When flows are triggered, Airflow writes task logs inside the worker pods, typically under the directory /opt/airflow/logs, which are surfaced in the Airflow UI.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   The first line shows the path of the generated log file inside the pod. To persist logs beyond the typical ephemeral pod lifecycle, this directory is mounted on Kubernetes PersistentVolumeClaims (PVC), often backed by NFS storage as a manual task as part of the Airflow deployment process. Accessing these logs externally requires connecting to the exact NFS export path backing the PVC, as logs aren’t typically directly stored on your local Kubernetes nodes. Within pods, you can run mount | grep logs to discover the actual NFS path backing the PVC. For example, the external storage location could look like: nfs-server.gelcorp.com:/srv/nfs/kubedata/airflow-airflow-logs-pvc-xxxxxx   Mounting this exact path on an external server or your workstation via NFS client tools enables you to browse and tail Airflow logs without pod access. In my lab:       How Logs Are Structured for Easy Identification   Airflow organises logs hierarchically using key=value directories such as: ->dag_id=my_first_dag ---->run_id=manual__2025-11-05T10:31:59.912269+00:00 -------->task_id=bash   This naming scheme uniquely identifies logs belonging to specific DAG runs and tasks, even when multiple flows operate simultaneously. Shell utilities may display these directory names enclosed in quotes due to special characters (like =), but the names are standard and intentional.   With this approach, users and admins can view logs for each run directly from the command line, providing robust control and insight into Airflow-managed workflows. Log messages are stored in standard JSON for flexibility.     Alternative Log Access Methods   The Airflow Web UI remains the primary and user-friendly method to view DAG and task logs in real-time. It aggregates logs and contextualises them with run metadata, filtering, and search.   You can also view pod logs with kubectl logs -n airflow airflow-worker-0 to see logs streamed from Airflow worker containers, which include some task execution output. This method can be helpful for quick debugging, but:   messages are not those displayed in the Airflow UI; these are are less granular than task log files it requires appropriate RBAC permissions to be granted it can be trickier to match messages with the tasks that generated them   2025-11-05 10:32:00.710584 [info ] Task execute_workload[0d8c4993-55b7-4aae-a0fa-81af04d5cc63] received [celery.worker.strategy] 2025-11-05 10:32:00.720534 [info ] [0d8c4993-55b7-4aae-a0fa-81af04d5cc63] Executing workload in Celery: token='eyJ***' ti=TaskInstance(id=UUID('019a5392-c8b7-7b00-9138-482da0a9f290'), task_id='bash', dag_id='my_first_dag', run_id='manual__2025-11-05T10:31:59.912269+00:00', try_number=1, map_index=-1, pool_slots=1, queue='default', priority_weight=2, executor_config=None, parent_context_carrier={}, context_carrier={}, queued_dttm=None) dag_rel_path=PurePosixPath('first_dag.py') bundle_info=BundleInfo(name='dags-folder', version=None) log_path='dag_id=my_first_dag/run_id=manual__2025-11-05T10:31:59.912269+00:00/task_id=bash/attempt=1.log' type='ExecuteTask' [airflow.providers.celery.executors.celery_executor_utils] 2025-11-05 10:32:00.771932 [info ] Secrets backends loaded for worker [supervisor] backend_classes=['EnvironmentVariablesBackend'] count=1 2025-11-05 10:32:02.464392 [info ] Task finished [supervisor] duration=1.7101506830003927 exit_code=0 final_state=success   Although tedious, you can also exec into the worker pods with kubectl exec -it -n airflow airflow-worker-0 to directly access and browse the logs directories inside pods. This method also requires appropriate access to be granted by your Kubernetes administrator.     Integration with observability tools   Another possible option is to collect log messages via Fluent Bit, Logstash, or similar tools for forwarding logs to a visualisation platform. With additional configuration, Airflow's task and component logs can be directed through these logging pipelines into tools like OpenSearch Dashboards, providing centralised log search, alerting, and dashboarding.   Depending on your existing logging infrastructure, it may be possible to configure your EFK/ELK stack to pick up Airflow logs by including the relevant namespaces and pod logs in the collection inputs. This approach enables integration of Airflow logs alongside other Kubernetes application logs, improving observability and monitoring.   If deployed, SAS Viya Monitoring for Kubernetes captures and indexes Airflow logs, but they are as limited as those coming from running a <code>kubectl logs</code> command, and not as granular as they are for SAS Viya log messages. The Fluent Bit configuration used by SAS Viya Monitoring for Kubernetes is targeted primarily at the SAS Viya namespaces and does not include Airflow's namespace in its logging filters. Take care if modifying; changes to the configuration are untested and may cause issues. Check latest documentation and support for extending log collection scopes before making changes.     Summary   External log access is important for Airflow observability in SAS Viya Kubernetes deployments. By identifying and mounting the correct NFS export path backing the airflow-logs PVC, you can browse logs from the command line without exec-ing into pods. The Web UI is great for monitoring interactive runs, and kubectl logs provides another complementary access method, but the ability to access them externally is helpful for users to monitor task progress quickly from the command line, and for administrators to perform maintenance, auditing, backups, and more on the aggregated logs. Integrating Airflow logs into observability streams further enhances log management and monitoring capabilities.     Find more articles from SAS Global Enablement and Learning here. ... View more

With the Stable 2025.06 version of SAS Viya, administrators now have access to a new asset for auditing platform usage: the User Activity Insights report. This report supersedes the existing User Activity Report and is part of a broader initiative to enhance the SAS Viya auditing framework. Let's take a closer look at this new report and see how it can help administrators more easily visualise user behaviour.   The User Activity Insights report is a new SAS Visual Analytics report designed to help administrators better understand how users are interacting with the SAS Viya platform. Unlike the existing User Activity Report, which is based on detailed audit records from the SystemData.AUDIT CAS table, this new report is built on activity records from the AUDIT_ACTIVITIES CAS table. Activity records are higher-level summaries of user behavior, offering an intuitive view of platform usage patterns.   The Audit service periodically copies selected audit and activity records from the audit database into CAS. The settings are configurable in SAS Environment Manager, where administrators can choose the types of records that are loaded, and the frequency at which they are loaded (and more!). These records are then used as the data source for the User Activity Insights report (and previously, the User Activity report).   The report is available to all members of the SAS Administrators group by default. It can also be copied and customised as required. It is located in SAS Content > Products > SAS Environment Manager > Dashboard Items, where its predecessor, the User Activity report, also currently lives until it is officially deprecated. The new report provides a more focused and useful view of user activity.    Let's take a look at some of the visualisations the new User Activity Insights report offers. It includes several tabs, each offering a different perspective on user behavior. Here are some examples:     User Activity Analysis   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   Active User Bar Chart: Displays the most active users based on the number of actions performed. Count of User IDs: Shows the total number of unique users active during the selected time window. Actions Done by Users: A horizontal bar chart showing which users performed which types of actions (e.g. read, create, update, delete). Users and IP Address Bubble Chart: Maps user activity to IP addresses, with bubble size indicating volume of actions.     Application Analysis     Application and Service Table: Lists services and their usage frequency. Top Actions per Application: A bar chart showing the most common actions across applications. Application Activity Line Chart: Tracks application activity over time to identify usage trends.     Service and Authorization Analysis     Audit Actions Related to Audit Type: A stacked bar chart showing successful and failed actions by type. Total Count of Success and Failure: Summary metrics showing the number of successful and failed actions, as well as unauthorised access attempts. Service and Application Filters: Allows filtering by service, application, and user ID.     Success/Failure Rate     Failure Over Time Area Chart: Shows the frequency of failed actions over time. Success Over Time Area Chart: Displays successful actions over the same time range. Interactive Filters: Filter by service name, application, or IP address.     Object Activity Analysis     Most Accessed Object Types: A bar chart showing which object types (e.g. reports, folders) are accessed most frequently. Top Accessed Objects by Type: A treemap showing the most frequently accessed individual objects. Top Actions on Objects: A bar chart showing the most common actions performed on objects. Success and Failure Donut Charts: Circular charts showing the frequency of successful and failed actions by object type and action.     Session Analysis Description     Session ID and Correlator Table: Displays a list of session IDs and correlators, helping administrators "trace" user sessions and match actions across services. Frequency of Application Bar Chart: Shows how often different applications were used during the selected time window, highlighting which tools are most actively used. In some cases, "missing" may appear when application metadata is unavailable. User ID Word Cloud: A visual representation of user activity, where the size of each user ID reflects the frequency of their actions. This provides a quick, intuitive view of the most active users. Frequency of Object Type Bar Chart: Highlights which object types (e.g. AuthorizationRules) are most frequently involved in user actions. Actions in Sequence Table: A chronological table showing the sequence of actions performed, including timestamps, action types, associated applications, and object types.   There are lots of options for drilling and filtering the displayed visuals and viewing the detailed data in the report.   If you're new to auditing in SAS Viya, check out my earlier post, which provides an overview of the original User Activity Report and the foundational concepts behind the Audit service.   With the addition of the User Activity Insights report, SAS Viya empowers administrators with deeper visibility into user behaviour and system usage. Watch this space for more features coming soon.     Find more articles from SAS Global Enablement and Learning here. ... View more

Using a local mirror registry to deploy and update SAS Viya is a common practice, with many organisations adopting secure, air-gapped or restricted network environments. Mirror registries offer control and security, but there are some things to consider when applying updates, especially around version management, tooling compatibility, and deployment orchestration. In this post, we’ll explore some common pitfalls and good practices when using a local mirror registry for SAS Viya deployment updates.     Keeping Everything in Sync   When using a local mirror for your SAS Viya deployment, you're responsible for mirroring required container images from cr.sas.com. The key challenge is keeping those in sync with the deployment assets for the version or release you want to update to, as well as syncing those with the correct version of sas-orchestration! If your deployment assets reference image versions that don’t exist in your mirror, or if your orchestration tool is out of sync with your assets, your deployment will fail.     Using the Right Image Versions   Each SAS Viya cadence release (e.g., 20250612.1749750470829) is tied to a specific set of container image versions. If your mirror registry is missing (or has an incorrect version of) one of these images, the deployment will hang or fail during orchestration.   There are some steps you can take to avoid this:   Use the viya4-orders-cli to fetch deployment assets for the exact cadence release you need; for instance, the release number displayed when you query available updates using the SAS Update Checker. In a dark site, you can periodically run the viya4-orders CLI on any machine (including client workstations or jump hosts) that can connect to ses.sas.com over the internet in order to retrieve the latest assets for a given cadence version. Update the mirror using the downloaded deployment assets to ensure the correct image versions are pulled and pushed to the mirror. Validate that all referenced images exist in your local registry before deploying using the mirrormgr CLI.     Mismatched Tool Versions   Tools like sas-orchestration and viya4-orders-cli are version-sensitive. The version of sas-orchestration used for the update must match the version required by the deployment assets, to avoid errors like this:   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   Download the version of sas-orchestration specified in the deployment assets (in $deploy/edu/sas-bases/.orchestration/images.yaml) to avoid a version mismatch and to continue with the deployment. To do so, pull it from cr.sas.com, tag it, and then deploy/update:   cloud-user@server:~/project/deploy/edu$ cat ~/project/deploy/edu/sas-bases/examples/kubernetes-tools/README.md | grep -A13 Prerequisites ## Prerequisites To run the sas-orchestration image, Docker must be installed. Pull the `sas-orchestration` image: ``` docker pull cr.sas.com/viya-4-x64_oci_linux_2-docker/sas-orchestration:1.142.1-20250429.1745959628012 ``` Replace 'cr.sas.com/viya-4-x64_oci_linux_2-docker/sas-orchestration:1.142.1-20250429.1745959628012' with a local tag for ease of use in the examples that will follow: ``` docker tag cr.sas.com/viya-4-x64_oci_linux_2-docker/sas-orchestration:1.142.1-20250429.1745959628012 sas-orchestration     Correctly specifying flags   When running the deployment command, you might be tempted to let the deployment tools figure out the latest available release by setting the --cadence-release flag as follows:   sas-orchestration \ deploy \ --namespace edu \ --deployment-data /license/certs.zip \ --license /license/license.jwt \ --user-content edu \ --cadence-name lts \ --cadence-version 2025.03 \ --cadence-release "" \   This won't always work in a mirrored environment. Why?   The tool will attempt to deploy the latest available release from SAS. If your mirror doesn’t contain that release, the deployment will fail.   A good practice is to specify the exact cadence release you mirrored (i.e. to match your downloaded deployment assets). Don't forget to add the --image-registry flag:   sas-orchestration \ deploy \ --namespace edu \ --deployment-data /license/certs.zip \ --license /license/license.jwt \ --user-content edu \ --cadence-name lts \ --cadence-version 2025.03 \ --cadence-release 20250711.1752209235788 \ --image-registry mirror.demo.com:80   Also, specifying the exact release required in your SAS Viya Orders CLI command takes things one step further in ensuring the same release number is used consistently.     Recommended Workflow   Download deployment assets: viya4-orders-cli -c ~/.viya4-orders-cli.env dep $order lts 2025.03 -p ~/. Download SAS Mirror Manager and mirror container images (using the new assets) to your local registry. Download the matching sas-orchestration version as required by the assets. Run deployment specifying the exact cadence version and release.     Final Thoughts   Using a local mirror registry for SAS Viya deployments gives you control, but it also requires some discipline. The key to success is version alignment: Match your images, tools, and assets, and avoid "" or unspecified versions. The deployment tools display log output, but the root causes of problems experienced during updates aren't always immediately obvious. It's a good idea to prepare appropriately, check pod logs in the case of issues, and validate everything after updating.   By following these practices, you’ll avoid the most common pitfalls and ensure smooth, repeatable deployments, even in air-gapped environments.     Find more articles from SAS Global Enablement and Learning here. ... View more

SAS Viya Monitoring for Kubernetes has introduced two features in version 1.2.36 (15th April 2025) aimed at simplifying deployment; auto-generating Ingress definitions and automatically specifying storage class references. These enhancements reduce the need for manual configuration, making setup easier and more efficient.   How Ingress is Used in SAS Viya Monitoring   Ingress resources in Kubernetes manage external access to services within the cluster, typically HTTP and HTTPS traffic. In SAS Viya Monitoring, Ingress facilitates access to web-based monitoring tools like Grafana and OpenSearch Dashboards (OSD). Properly configured Ingress ensures that users can securely and efficiently interact with these interfaces.   There are two primary approaches to Ingress configuration:   Host-Based Ingress: Each application is accessed via a unique subdomain. Example: - Grafana: https://grafana.gelcorp.com/   - OpenSearch Dashboards (OSD): https://osd.gelcorp.com/     Path-Based Ingress: A single domain is used, with applications differentiated by URL paths. Example: - Grafana: https://monitoring.gelcorp.com/grafana   - OpenSearch Dashboards (OSD): https://monitoring.gelcorp.com/osd     The choice between host-based and path-based Ingress depends on your organisation's policies. Detailed guidance on configuring Ingress for SAS Viya Monitoring is available in the official documentation.   Auto-Generated Ingress Definitions   Previously, configuring Ingress resources required manual editing of YAML files. With this new feature, users can automate the creation of these definitions by setting environment variables, preferably by including them in the $USER_DIR/user.env file. See the Pre-Deployment topic in the Help Center documentation for information on the $USER_DIR directory and more information about the SAS Viya Monitoring customisation process. The system supports both host-based and path-based Ingress configurations, aligning with common deployment scenarios.   To enable auto-generated Ingress definitions:   Set the environment variable AUTOGENERATE_INGRESS=true. Specify your Ingress type by setting ROUTING to either host or path. Specify the hostname to be used in the URLs by setting the environment variable BASE_DOMAIN. This is required. Apply your configuration as usual, and the system will generate the appropriate Ingress definitions automatically.   If you provide your own TLS certificate files, it's also possible for the autogeneration process automatically create corresponding Kubernetes secrets needed by the Ingress resources. Two further variables are required for this:   Set INGRESS_CERT to point to the location of the TLS certificate file. Set INGRESS_KEY to the TLS key file.   How Storage Classes Are Used in SAS Viya Monitoring   Persistent storage is crucial for retaining monitoring data, logs, and metrics across pod restarts and deployments. Kubernetes uses Persistent Volume Claims (PVCs) to request storage resources, and storage classes define the types of storage available for these PVCs.   In SAS Viya Monitoring, various components require persistent storage to function correctly. For instance, Prometheus needs storage to retain metric data, and OpenSearch requires storage for log data. Configuring appropriate storage classes ensures that these components have access to reliable storage resources.   Key considerations for storage configuration include:   Durability: Ensure that data persists beyond the lifecycle of individual pods by using persistent volumes external to the Kubernetes cluster. This approach safeguards data integrity during scaling or pod failures. Performance: Select storage classes that meet the performance requirements of your monitoring workloads, considering factors like input/output operations per second (IOPS) and latency. Capacity Planning: Allocate sufficient storage capacity to accommodate the volume of monitoring data generated, with appropriate provisions for future growth.   For comprehensive information on storage requirements and configuration in SAS Viya Monitoring, refer to the official documentation.   Automatic Storage Class References   Specifying storage classes for PVCs traditionally involved modifying YAML files manually. The latest update introduces an automated approach, generating the necessary storage class references based on predefined environment variables, preferably by including them in the $USER_DIR/user.env file. See the Customization topic in the Help Center documentation for information on the customisation process. While this simplifies deployment for standard configurations, manual customisation remains an option for more complex setups.   To enable automatic storage class references:   Set the environment variable AUTOGENERATE_STORAGECLASS=true. Define your preferred storage class by setting STORAGECLASS to the appropriate value for your environment. It is important to note that the storageClass specified must exist; SAS Viya Monitoring will not create any storageClass resources. The system will apply this storage class to all relevant PVCs at deployment time without requiring manual edits.   Next Steps   These new capabilities provide a streamlined way to configure Ingress and storage in SAS Viya Monitoring. By reducing manual effort and automating common setup tasks, they make deployment more efficient while still allowing for customisation where needed. Refer to the the Help Center for more information about customising the process to suit your needs.   To get started, ensure you have yq version 4.32.2 or later (which will be a requirement for future releases of SAS Viya Monitoring for Kubernetes). Detailed instructions, including configuration examples, can be found in the documentation.   My thanks to Greg Smith for his contributions to this post.     Find more articles from SAS Global Enablement and Learning here. ... View more

The SAS Viya Orders CLI, the command-line tool for managing SAS Viya software order assets, has recently introduced a new feature that enhances control and precision when downloading deployment assets. As of the latest version (1.7.0, released February 2025) available from the GitHub repository, users can now specify a cadence release value when downloading deployment assets. This addition addresses a use case you may have encountered when deploying or updating your software. Let’s check out how it works and why it matters.   The Problem: Hidden Patches in the Same Version Number   Imagine this scenario: On March 27th, a new SAS Viya version is released. You download the deployment assets and deploy them to a test environment. After running your tests, everything looks good, and you greenlight the deployment to production on March 30th.   Between March 27th and March 30th, SAS releases several patches for that version. The version number remains unchanged, and on the surface, there’s no obvious indication that the assets have been updated to include those new patches. You download the artifacts again on March 30th, expecting the same package you tested, only to find that the patched version is different. This discrepancy might go unnoticed until the production system behaves in an unexpected manner or generates some form of error.   The Solution: Specifying cadence release in the CLI   The new feature in the SAS Viya Orders CLI allows users to explicitly specify a cadence release, which denotes the 'patch version', when downloading deployment assets. This means you’re no longer constrained to downloading the latest patched version of SAS Viya, which was previously the default behaviour. Instead, you can pinpoint the exact release you want, matching what you tested, and avoid surprises in production.   This functionality mirrors capabilities in the SAS Viya Deployment Operator and sas-orchestration tool, where users can define a specific cadence release to ensure consistency across environments. By bringing this precision to the SAS Viya Orders CLI, SAS empowers users to maintain control over their deployment process, aligning with real-world needs where testing and production must stay in sync.   How It Works   The updated CLI now includes an option to specify the cadence release alongside the cadence name and version when using the deploymentAssets command. The exact syntax might look something like this (check the latest README.md for confirmation):   viya4-orders-cli deploymentAssets / <order-number> / <cadence-name> / <version-number> / <cadence-release-number>   Here’s a breakdown:   <order-number> : Your specific SAS Viya order number. <cadence-name>: The release cadence (e.g., Stable, LTS). <cadence-version>: The version number (e.g., 2025.03). <cadence-release-number>: The new value that identifies the exact patch level or release ID you want.   For example, if you tested 2025.03 with a cadence release of 20250327*, you could ensure you download that exact artifact for production, even if patches have since incremented the release to, say, 20250330.   * Simplified example only; actual release numbers have an epoch date-time value like 20250326.1721414421837.   Why This Matters   This new feature tackles several key challenges:   You can now guarantee that what you tested is what you deploy, avoiding the risk of untested patches appearing in production environments. No more guessing whether a release has been patched. You decide which artifact to pull based on your needs. For organisations with strict compliance requirements, this ensures you can track and replicate/automate the exact software state across deployments. You can still get the latest patch version by leaving cadence release unspecified.   This level of control eliminates the disconnect between testing and deployment, making the process smoother and more predictable.   Final Thoughts   The addition of the cadence release option in the SAS Viya Orders CLI is a useful feature for anyone managing SAS Viya deployments. It addresses a real-world need, balancing the need to stay up-to-date with software patches with the stability required for production environments.     Find more articles from SAS Global Enablement and Learning here. ... View more

SAS Viya continues to evolve, offering organisations flexibility in how they manage software updates. As part of this ongoing refinement, changes have been introduced to update paths for the Long-Term Support (LTS) cadence. These adjustments impact how customers can perform updates and maintain their SAS Viya LTS environments. By staying informed and aligning update strategies with business needs, customers can maximise the value of SAS Viya while minimising potential issues and disruptions.   Reviewing The Two Update Cadences   Long-Term Support (LTS)   Designed for stability and predictability. Updates are less frequent but include critical fixes and security patches. Ideal for enterprises that require a consistent and well-tested environment over an extended period.   Stable   Provides access to the latest features and improvements more frequently. Allows customers to take advantage of innovations as they become available. Suitable for organisations that prioritise agility and staying up to date with the latest advancements in SAS Viya.   What’s Changing?   Recent adjustments to the update process affect how customers can move to newer LTS versions. These changes aim to reduce the likelihood of problems and ensure smoother transitions when updating software.   Update paths are documented for supported versions: The SAS Viya Platform Administration Guide's Versions in Standard Support page explicitly lists support levels for versions of SAS Viya as well as supported update pathways. Check the documentation prior to each update, to ensure you are aware of latest versions released as well as those dropped from standard support.     Example upgrade paths for 2025.02    Select this image to see a larger version. Mobile users: To view the image, select the "Full" version at the bottom of the page.   LTS Update Paths: A new policy has been officially adopted, requiring customers to follow a sequential update process when updating LTS versions. Skipping LTS releases is no longer supported. For example, customers on LTS 2023.03 must update to LTS 2023.10, then LTS 2024.03, and then LTS 2024.09 in sequence. Each update must follow pre-update checklists, review deployment notes, and assess the What’s New documentation to ensure compatibility and a smooth transition.   Kubernetes Version Considerations: Customers must verify Kubernetes version compatibility when updating SAS Viya. Certain Kubernetes version required by LTS releases may be unsupported or on extended support for a given cloud provider.   What This Means for Customers   Organisations should review their update strategies in light of these changes. Whether your priority is long-term stability or continuous feature updates, understanding the refined update paths will help ensure a seamless experience with upgrading your SAS Viya deployment.   For full details on these changes, including technical requirements and best practices, refer to the updated documentation.   My thanks to Edie Jeffreys for her contributions.       Find more articles from SAS Global Enablement and Learning here. ... View more

There are a number of useful features, settings and tools that administrators can utilise to audit user activity in SAS Viya 3.x. In this post, we will explore the options available for the auditing of report actions in SAS Visual Analytics.   Important   Before reading on, it should be stressed that there may be a negative impact on system performance as a result of modifying the auditing configuration to capture more detailed records. Broadening the scope of the auditing configuration will generate considerable amounts of data, which will result in a level of slowness, as well as increased disk consumption. If you encounter poor performance after changing the configuration, you may need to reduce the scope by limiting the applications, actions or resources for which to capture audit records.   Overview   Auditing in SAS Viya is based on the SAS Operations Infrastructure, which is based on SAS's event-driven architecture. The event framework uses RabbitMQ as the message broker for processing all events generated by SAS Viya services.   The default behaviour is to capture four basic actions (create, read [failure], update and delete) on report objects. This can be modified using the configuration properties for the Audit service's sas.audit.record configuration instance in SAS Environment Manager.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   When processing an event, the Audit service uses the properties defined here to determine whether it is captured in the audit records.     Configuration properties must be defined in a specific pattern. The following are valid options for configuring the audit service to capture records on a per application basis:   application-name.enabled application-name.event-action-type.enabled application-name.event-action-type.state   where application-name refers to the service name (for example, reports, identities or compute) and event-action-type is one of the action properties from an internal event action registry. State refers to the success or failure of the event action.   Additionally, the configuration properties can be set to capture actions based on entry type rather than by application. For example, security actions can be captured by setting security.action.login.state=all in the type property.   Audit records are written to the SAS Infrastructure Data Server. The data is not loaded to the CAS table that drives the pre-canned audit reports in Environment Manager until the Operations Infrastructure's genAudit task is run (every 2 hours by default). We can use the sas-admin CLI's audit plugin to view the audit records in PostgreSQL immediately. Keep reading for some examples.   Records older than 7 days in the datamart are archived to a specified location on disk. The sas.audit.archive configuration instance determines the rules for the archiving of audit records.   Auditing Report Access   How can we tell what reports our user (Ahmed) opened? Well, remember that audit records are only created for failed read attempts by default due to the setting of resource.action.read.state=failure in sas.audit.record. We can update this value to all to capture successful as well as failed attempts for all resources, but note that this is generally not recommended due to the impact on performance as a result of the amount of data generated. I set this option in my lab for the purpose of this demo, as I am the only user and it is a throwaway environment. After doing so, I ran the CLI command again, this time filtering for entries created by the reports application only. /opt/sas/viya/home/bin/sas-admin --output text audit list --user-id Ahmed --application reports   ID Time Stamp Action State User ID Application URI 94a2d4ba-aefc-4d71-9961-04ada7274e91 2019-10-02T02:31:05.253Z read success Ahmed reports /reports/reports/cbf97b0a-457d-4b4f-8913-547e0cdf390c b40b1ecc-8cd9-4733-99a7-c3e63e06adaf 2019-10-02T02:30:53.703Z read success Ahmed reports /reports/reports/cbf97b0a-457d-4b4f-8913-547e0cdf390c e57a7972-4b5c-4df9-90a6-ed038a4e74cb 2019-10-02T02:30:52.806Z read success Ahmed reports /reports/reports/cbf97b0a-457d-4b4f-8913-547e0cdf390c c71d2094-d18e-4cee-8e03-6e8c7505d070 2019-10-02T02:30:10.382Z read success Ahmed reports /reports/reports/cbf97b0a-457d-4b4f-8913-547e0cdf390c cb10dfb4-bf7d-40f8-a709-456abff091d9 2019-10-02T02:30:00.277Z read success Ahmed reports /reports/reports/cbf97b0a-457d-4b4f-8913-547e0cdf390c 59735b62-d8f4-4f91-9a15-b447f76bb483 2019-10-02T02:16:54.625Z read success Ahmed reports /reports/reports/cbf97b0a-457d-4b4f-8913-547e0cdf390c   Interestingly, there are multiple read entries for the same report. The reason is that these reads represent more than just the action of opening of a report. In fact, some of these records are created when a report is selected in the Content area of SAS Environment Manager, or when thumbnails for the reports are rendered in the list of Recent reports in Report Viewer and SAS Drive (documented in SAS Note 62355). To identify the record that represents the actual opening of the report, we need another option in our command. /opt/sas/viya/home/bin/sas-admin --output text audit list --user-id Ahmed --application reports --details   ID Time Stamp Type Action State Description User ID Application Remote Address URI 94a2d4ba-aefc-4d71-9961-04ada7274e91 2019-10-02T02:31:05.253Z resource read success Ahmed reports 192.168.1.2 /reports/reports/cbf97b0a-457d-4b4f-8913-547 b40b1ecc-8cd9-4733-99a7-c3e63e06adaf 2019-10-02T02:30:53.703Z resource read success Ahmed reports 192.168.1.1 /reports/reports/cbf97b0a-457d-4b4f-8913-547 e57a7972-4b5c-4df9-90a6-ed038a4e74cb 2019-10-02T02:30:52.806Z resource read success Ahmed reports 10.96.17.168 /reports/reports/cbf97b0a-457d-4b4f-8913-547   Adding the details option displays IP addresses in the output. The one that represents the report open is the one that containing the IP address of the client machine. In this example, that would be 10.96.17.168 (the record on the bottom).   We can use the CLI to display more detailed information about an audit record by running the audit plugin's show-info command. /opt/sas/viya/home/bin/sas-admin --output text audit show-info --id 94a2d4ba-aefc-4d71-9961-04ada7274e91   ID 94a2d4ba-aefc-4d71-9961-04ada7274e91 Time Stamp 2019-10-02T02:31:05.253Z Type resource Action read State success User ID Ahmed Remote Address 192.168.1.2 Trace ID 24b3c0cac3add8e3 Application reports URI /reports/reports/cbf97b0a-457d-4b4f-8913-547e0cdf390c Folder /Products/SAS Visual Analytics/Samples Report Name Retail Insights HTTP Status Code 200 HTTP Method GET   As indicated in the SAS Note, only reports under the /Products and /Public folders produce audit records by default. Follow the steps in the note to modify the configuration to capture audit records for reports in all locations.   Auditing Report Actions   For auditing other report actions, such as filtering, exporting, printing, sharing reports, it's a little more complicated.   Filtering   Applying filters to your report will produce more audit records similar to the below from the reportData application, but there is currently no simple way to determine exactly how the report was filtered.   81d4100b-92e1-4dc2-9464-1fe102a062a9 2019-10-02T05:07:24.771Z create success Ahmed reportData /reportData/jobs/0c3691a3-7c55-4774-af19-684a1eda8641_c36 de376ccb-30ee-4ec5-90d7-dfad2b548d19 2019-10-02T05:07:25.901Z create success Ahmed reportData /reportData/jobs/0c3691a3-7c55-4774-af19-684a1eda8641_c35 8a632ec8-e145-4c1e-ad46-69e377270ecf 2019-10-02T05:07:25.943Z read success Ahmed reportData /reportData/results/0c3691a3-7c55-4774-af19-684a1eda8641_0c3691a3-7c55-4774-af19-684a1eda8641_c35/files/dd27639.csv b33a90b2-5ba1-4cda-aeb7-aeb42356f358 2019-10-02T05:07:25.949Z read success Ahmed reportData /reportData/results/0c3691a3-7c55-4774-af19-684a1eda8641_0c3691a3-7c55-4774-af19-684a1eda8641_c35/files/dd27639index.csv   Exporting   The transfer service is responsible for handling the export of content. Therefore, assuming resource.action.read.enabled is set to its default value of failure, we need to turn on auditing for the transfer service. We can do so by adding these new entries to the application property in the sas.audit.record configuration instance.     This will capture all CRUD events (including successful reads) for the transfer service only. Now, when exporting a report from SAS Environment Manager's Content area, audit records similar to the below are captured.   51f68e79-000c-4f82-8015-dcaebdc6552e 2019-09-25T13:25:52.719Z resource create success sasadm transfer 10.96.10.154 /transfer/exportJobs/7f1bbfc7-fdfd-4f09-8527-97fb071746e0 ... 82c18db3-414d-4cde-8b0a-6e76cb3fc315 2019-09-25T13:25:54.823Z resource read success sasadm transfer 10.96.10.154 /transfer/packages/85d85fac-f7f4-4d34-9e74-31401b34caf1   (Note that in this case, the User ID is sasadm. Remember that only administrators can export reports.)   Conveniently, the last column in the second record displays the URI to the package file. We can use the sas-admin CLI's transfer plug-in to display its contents.   /opt/sas/viya/home/bin/sas-admin transfer show-contents --uri /transfer/packages/85d85fac-f7f4-4d34-9e74-31401b34caf1   Name ContentUri ContentMediaType Retail Insights /reports/reports/cbf97b0a-457d-4b4f-8913-547e0cdf390c application/vnd.sas.report Samples /folders/folders/8af48321-d89c-4522-9557-079bce15edce application/vnd.sas.content.folder SAS Visual Analytics /folders/folders/f865778e-db27-4b4a-be3d-6322bd4ff81d application/vnd.sas.content.folder Products /folders/folders/1d8b18bf-853b-4a57-8600-c9bd37cf49ae application/vnd.sas.content.folder   In this example, the package contains the Retail Insights report and its corresponding SAS folder structure.   When importing packages, the transfer service will generate similar events that are also captured as audit records.   What about when exporting data to Excel from within a VA report? This will require successful reads to be turned on for the reportData application (reportData.action.read.state=all). We will then get the following records.   d81d2344-35ff-4bc8-af81-6ad957f37271 2019-10-02T03:34:32.592Z read success Ahmed reportData /reportData/results/310e8def-27a1-4f32-8676-05ce504d24b3_c1635e4d-618c-4968-9d9c-2d624fcb4279/exportFiles/dd431.xlsx ... 6778fbdb-2cce-4844-8fd1-beca87ce4cec 2019-10-02T03:34:32.445Z create success Ahmed reportData /reportData/jobs/c1635e4d-618c-4968-9d9c-2d624fcb4279   The read entry lists the Excel file, which is great, but notice anything that links it to the create entry? The URI contains a reference (after the underscore) to the create job entry.   Printing   Printing in Visual Analytics or Report Viewer generates a PDF of the report which the user can then print. This action results in the following audit records, indicating the name of the PDF that was produced.   4ebe39db-dd57-4c20-a61e-506f70602a9e 2019-09-25T11:34:14.826Z resource read success Ahmed reportRenderer 10.96.10.154 /reportRenderer/reports/3BIEOARCKYIRJU4KKR67XKTXDQAU3MWA/Retail%20Insights%20on%2009-25-2019.pdf 5fe92d47-ff6f-4134-880e-234985589c6d 2019-09-25T11:34:14.755Z resource create success Ahmed reportRenderer 10.96.10.154 /reportRenderer/reports/3BIEOARCKYIRJU4KKR67XKTXDQAU3MWA   Other Client-side Actions   Many other user actions performed in SAS Visual Analytics are client-side, meaning they do not interact with other services in a meaningful way (in this context). As such, they do not currently generate audit records in a manner that can be surfaced appropriately in SAS Viya 3.x. Auditing capabilities are constantly being improved, though, particularly in later versions of SAS.  Conclusion   Many organizations perform auditing for regulatory compliance or to report on security and usability aspects of the platform. Viya goes some way to meet basic auditing requirements out of the box, and provides the flexibility to expand the auditing scope with relative ease - BUT, do not forget the potential performance implications, as they can be significant. Keep an eye out for further improvements in auditing functionality in future releases. In later posts, we will explore auditing other areas of the SAS Viya platform.   For more information, refer to the official documentation.   Thanks to Eric Bourn, Todd Braswell and Anant Patil for their invaluable contributions to this post.   Thank you for reading. I hope the information provided in this post has been helpful. Please leave a comment below to share your own experiences.     Find more articles from SAS Global Enablement and Learning here. ... View more

When deploying SAS Viya, the default behavior schedules both stateful and stateless workloads across any available node pools. Generally, this setup is ideal, as Kubernetes is tasked with efficiently allocating pods to the appropriate nodes. However, there may be some edge cases where stricter scheduling of workloads is required. For example, it may be desirable to segregate stateful and stateless workloads to better utilise resources across different node pools, such as in situations where solutions like SAS Visual Investigator (with pods that demand more storage or memory) are deployed across node pools of varying sizes. In this post, we'll talk about confining stateful workloads to stateful node pools, and stateless workloads exclusively to stateless node pools.   Ensuring Workload Separation   The primary challenge lies in controlling where different types of pods are scheduled. By default, both StatefulSet (stateful) and Deployment (stateless) pods tolerate both stateful and stateless node classes. This flexibility can sometimes result in stateful pods being scheduled on stateless nodes, which might not have appropriate resource allocations, and could potentially lead to throttling or pod eviction during high activity from stateless pods.   Two approaches can be taken to address this challenge.   Setting up required node affinity   Node affinity allows us to define rules that specify where pods should be scheduled, based on the labels applied to nodes. This approach offers more granularity and flexibility, allowing for strict scheduling of workloads on particular nodes. In this case, we can switch the affinity rules from 'preferred' to 'required', as covered by my colleague Raphaël Poumarede in his post.   Begin by labeling the nodes to distinguish between stateful and stateless node pools.   kubectl label nodes workload.sas.com/class=stateful kubectl label nodes workload.sas.com/class=stateless   Add the following node affinity rules to StatefulSets to ensure these pods are scheduled only on stateful nodes.   spec: template: spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: workload.sas.com/class operator: In values: - stateful   Similarly, ensure that stateless pods (Deployments) are scheduled only on stateless nodes by modifying the YAML templates as follows:   spec: template: spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: workload.sas.com/class operator: In values: - stateless   Rebuild and redeploy to apply these changes.   Modifying tolerations   Tolerations work with taints applied to nodes, determining which pods are allowed to run on nodes with specific taints. By default, stateful and stateless pods tolerate both node classes. Modifying tolerations provides an alternative, simpler method of enforcing workload placement.   We can modify (remove) the tolerations in the YAML files that define the pods to achieve segregation of stateless and stateful workloads.   Here’s a sample patch to remove the stateful toleration from all Deployment resources to confine stateless pods to nodes optimised for stateless workloads.   --- apiVersion: builtin kind: PatchTransformer metadata: name: remove-stateful-toleration patch: |- - op: remove path: /spec/template/spec/tolerations value: - effect: NoSchedule key: workload.sas.com/class operator: Equal value: stateful target: group: apps kind: Deployment version: v1   An update to kustomization.yaml is also required:   ... transformers: - remove-stateful-toleration.yaml # Path to your patch file   Once applied, this patch will ensure that deployment resources are no longer scheduled on stateful nodes, improving the separation of workloads and enforce the changes across the environment.   Choosing the right approach   Both approaches outlined here offer effective methods for ensuring that stateful and stateless workloads are scheduled on the appropriate node pools in SAS Viya deployments. Other types workloads can also be segregated as desired. Note, however, that the default behaviour of scheduling across any available node pool is usually perfectly appropriate.   Required affinity provides more flexibility and control to enforce scheduling rules based on node labels, which can be easily adjusted if new node pools are added or removed. There is, however, an added administration overhead of maintaining node labels.   Modifying tolerations offers a simpler method for restricting pod placement by removing tolerations for nodes to repel unsuitable pods, but there is somewhat less flexibility, such as in handling changes to node pools.   Either method (or a combination of both methods) can be used to ensure stricter workload distribution in your SAS Viya environment with pods more efficiently distributed across node pools.   My thanks to Raphaël Poumarede, Harshit Soni and Henrique Lima for their contributions.       Find more articles from SAS Global Enablement and Learning here. ... View more

Auditing capabilities available in SAS Viya are essential for understanding user behavior and ensuring that resources are used appropriately. In this post, we'll demonstrate a method of tracking and viewing user access to SAS Visual Analytics reports using audit data in SAS Viya.   In SAS Viya, the Audit service is responsible for the capture and storage of records relating to user activity. By default, the Audit service does not capture successful 'read' events, so to capture information about who accessed a report, a simple change is first required to the sas.audit.record configuration instance in SAS Environment Manager's Configuration area.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   The audit.recording.level property can be adjusted to specify which events to capture. To enable the capture of successful 'reads', change the audit.record.level to a setting of High.     Note, though, that this can significantly increase the volume of data that is captured and stored. Your mileage may vary, but it’s important to balance capturing detailed information with managing storage requirements. One option is to define the service.list property, where you can specify a comma-separated list of application/service names for which successful 'read' events will generate audit records. It might also be prudent to adjust the retention periods for audit records.   Once that's done, the next time a user accesses a report, an audit record is created. In fact, many audit records are created. Suppose a user has viewed the sample Warranty Analysis report. The sas-viya --output text audit list --details command will return something like:   ID Time Stamp Type Action State Description User ID Application Remote Address Administrative Action URI 312c1b34-59b8-4fd9-97c8-016f9cb2ded2 2024-05-23T06:17:02.788Z resource read success Ahmed reports 127.0.0.1 true /Products/SAS Visual Analytics/Samples/Retail Insights.report 31d36d6b-6867-4840-b044-80b09a76c043 2024-05-23T06:17:02.69Z resource read success geladm reports 127.0.0.1 true /Products/SAS Visual Analytics/Samples/Warranty Analysis.report ceb2fadb-0bf2-4720-8e41-1eebb69a42c1 2024-05-23T06:17:02.596Z resource read success geladm reports 127.0.0.1 true /Products/SAS Visual Analytics/Samples/Water Consumption and Monitoring.report 4174fd1e-e0c9-498d-9394-a7ba955d58b1 2024-05-23T06:17:02.512Z resource read success geladm reports 127.0.0.1 true /Products/SAS Visual Analytics/Samples/Warranty Analysis.report 2841747c-fcdc-42b3-97cb-38bbcacd2bbd 2024-05-23T06:17:02.415Z resource read success geladm reports 127.0.0.1 true /Products/SAS Visual Analytics/Samples/Retail Insights.report c8e2e71c-dc49-4acf-bef4-42f57f27d1d1 2024-05-23T06:17:02.33Z resource read success geladm reports 127.0.0.1 true /Products/SAS Visual Analytics/Samples/Warranty Analysis.report bd9a5747-3871-470c-8010-3cbfe4c87ef1 2024-05-23T06:17:02.22Z resource read success geladm reports 127.0.0.1 true /Products/SAS Visual Analytics/Samples/Warranty Analysis.report 0540029e-08fe-4e59-9383-14cd2e7e40e7 2024-05-23T06:17:02.104Z resource read success Ahmed reports 127.0.0.1 true /Products/SAS Visual Analytics/Samples/Warranty Analysis.report 1417b61c-657a-4015-87d8-1b77a15bcb69 2024-05-23T06:16:42.226Z resource read success Ahmed reports 10.42.1.39 false /Products/SAS Visual Analytics/Samples/Warranty Analysis.report   Note that not only does Warranty Analysis appear multiple times as a successful read event, but other reports that we didn't open are also listed. This is because a "read" action is more than just the act of opening a report in SAS Visual Analytics. For example, when report thumbnails are rendered, these are also registered as "read" actions for the user, even if that user never actually opened the reports at that time.   How then do we differentiate between "reads" when a user physically opens a report and these other "reads"? To do so, we can use the RemoteAddress field (which is shown when the --details flag is added to the CLI command). The report that uniquely identifies the actual user action of opening the report contains the the podIP of the sas-folders pod as the value for the RemoteAddress field. We can get that with the command: kubectl get pod -l app=sas-folders -o custom-columns=NAME:metadata.name,IP:status.podIP   In my lab environment, that matches the last record displayed in the output above. The CLI command used to query the audit data could be modified to include a filter such that RemoteAddress equals the IP address of your sas-folders pod. Note that this behaviour is slightly different to the behaviour in SAS Viya 3.x.   Auditing the usage and access of SAS Visual Analytics reports is a critical aspect of managing and securing your SAS Viya environment. By leveraging the detailed audit data available, you can effectively monitor report access and usage and swiftly address any issues that arise. Implementing robust auditing practices not only aids in compliance but also promotes accountability and transparency within your organization.   For more information on auditing in SAS Viya, refer to the official SAS documentation.     Find more articles from SAS Global Enablement and Learning here. ... View more

In the world of monitoring and observability, the ability to detect and respond to issues promptly is paramount. Alertmanager and Grafana, both deployed with SAS Viya Monitoring for Kubernetes, possess alerting capabilities. While both serve the purpose of alerting, they have distinct features and approaches that cater to different needs. In this blog post, we'll endeavour to compare Alertmanager and Grafana by their alerting functions and provide some points to consider for when you might want to use one over the other.   Alertmanager's key features   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   Alertmanager is an open-source alerting tool that works seamlessly with Prometheus for a comprehensive metric-based monitoring and alerting toolkit. It is designed to handle alerts sent by client applications such as Prometheus and then manage these alerts by grouping, deduplicating, and routing them to various integrations like email, PagerDuty, Slack, and Microsoft Teams. Both are deployed together with SAS Viya Monitoring for Kubernetes. Since Alertmanager is tightly integrated with Prometheus, it leverages PromQL, Prometheus' powerful query language for defining alerting rules. This makes it easy to create sophisticated alerting conditions based on Prometheus metrics.   Alerts are created as PrometheusRules, and evaluated against metric data stored in Prometheus. Rules use YAML; easy to use, edit, store and deploy. Alertmanager allows users to define complex alert routing and inhibition rules based on labels attached to alerts. This flexibility enables fine-grained control over which alerts are sent to which receivers. For more details on how the routing process works, check out thisearlier post. Alertmanager provides a silence mechanism that allows users to temporarily mute specific alerts or entire groups of alerts.; useful when dealing with issues and addressing underlying causes. Alerts can also be inhibited in advance during planned outages, such as during maintenance activities. Alertmanager can be deployed in a highly available configuration to ensure reliability. It supports clustering and replication for fault tolerance.   Grafana's key features     Grafana is a popular open-source platform for monitoring and observability that offers a wide range of features, including visualization, dashboarding, and alerting. While Grafana is primarily known for its visualization capabilities, it also provides robust alerting functions. Older versions of Grafana required alerts to be created on dashboards, which was quite restrictive. While this is still possible, the Alerting facility that is now included offers much more comprehensive and flexible capabilities.   Grafana allows users to define alerting rules visually using its intuitive web interface. Users can set thresholds, conditions, and notification channels for triggering alerts directly in the UI (no need to write YAML or run kubectl commands). Grafana's alerting feature seamlessly integrates with its visualization capabilities. Users can create alerts directly from dashboards and visualize alerting states alongside other metrics. Grafana supports a variety of data sources; not just Prometheus. This flexibility enables users to create alerts based on data from different monitoring systems. Useful if other existing tools are already in use. Grafana keeps a history of triggered alerts, making it easy to track the lifecycle of alerts over time. Additionally, users can annotate graphs with alert events for better context.   Comparison   Authentication   Grafana requires users to log in with a username and password. Alertmanager has no such authentication built in; if the URL is accessible, anyone can use it. There are other ways to 'control' access to Alertmanager, but they need careful planning and consideration. Here is one example. Certainly, none are as convenient as Grafana's simple login prompt.   Ease of Use   Grafana's web interface provides an intuitive way to create and manage alerts (visually), making it more accessible to users who are not familiar with Prometheus' query language. On the other hand, Alertmanager's configuration may require more expertise due to its reliance on Prometheus' query language.     Default alerts   Prometheus includes a number of pre-defined alerts for Kubernetes applications when SAS Viya Monitoring for Kubernetes is deployed. Very useful for alerting on generally common issues, and no additional setup is required beyond defining notification channels for them. Grafana conveniently also displays these alerts, but interacting with them (e.g. silencing them when they fire) is a little less intuitive for these than it is for Grafana-managed alerts.     Integration   Both AlertManager and Grafana offer integration with various notification channels such as email, Slack, and PagerDuty. However, Grafana's integration capabilities extend beyond alerting to include visualization and dashboarding (Grafana is first a monitoring tool after all). There are also options for integrating the two together; for instance, to send Grafana-managed alerts to Alertmanager when they begin firing.   Scalability   Alertmanager can easily be scaled horizontally for high availability to handle large volumes of alerts; in fact, it's designed to be clustered. Grafana's scalability depends on the underlying data source it's connected to, but it can also scaled with a bit of effort. With SAS Viya Monitoring for Kubernetes, though, both are deployed as standalone instances by default.   Customisation   Alertmanager provides more fine-grained control over alert routing and inhibition rules, making it suitable for complex alerting workflows. It also offers options to customise nearly every aspect of the alert notifications. Grafana, on the other hand, offers greater flexibility in terms of visualization and dashboard customization, which Alertmanager lacks.   Conclusion   Both AlertManager and Grafana are powerful tools for alerting in the context of monitoring and observability. The choice between the two (or both!) depends on factors such as familiarity with Prometheus, integration and security requirements, and needs for advanced alerting features. Grafana may generally be considered more convenient for administrators with basic alerting requirements or whom are new to observability tools, whereas the expertise required to best leverage Alertmanager's powerful capabilities may be better suited to organisations that already use and have experience with it and related tools (e.g. Prometheus). Ultimately, organisations should evaluate their specific use cases and requirements to determine which tool best fits their needs. ... View more

The event-driven architecture in SAS Viya 3.5 (and 3.4) offers the capability for effective auditing of system resources, report activity, data access, and more. In this post, we’ll revisit the options available for auditing user access and highlight important caveats and considerations. This revised version of an earlier article has been updated to account for some changes in the underlying processes to provide a clearer picture.   CAUTION: Important considerations   Before adopting the approach to auditing user sessions documented in this post, beware these important cautions about using the EV Datamart data captured by the SAS Operations Infrastructure for this purpose: The data is not originally intended as 'audit' and it does not contain data integrity and security features that true audit data should. The data is derived from different data sources in which information is recorded differently, and matching user sessions with this data can be imprecise. It is not suitable for real-time or near-real-time purposes due to (potentially significant) latency in data being written to disk. Matching session activity like logons and logoffs using the session ID may not always work as expected. While the session ID is generally unique, it is not guaranteed to be unique. There are a small number of situations and edge-cases where a given session ID might be reused. SAS Viya 3.5 versions released in December 2023 (and later) address an issue where the session signature was not recorded for a login event, resulting in errors with the AUTHENTICATIONS table data. The information outlined in this post is a potential workaround for simplifying the process for auditing user session, but the caveats listed above must be noted as they present some potential challenges with this approach. Proceed with caution.   Information about logins and user sessions is captured by the Audit service in the AUDIT.security_audit tables in the SAS Infrastructure Data Server. The default behaviour, as defined in the security type properties in the sas.audit.record configuration instance, is to capture read failures only on security actions.   Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.    When trying to audit user login attempts, the login event provides information about (successful and failed) login attempts coming from the SASLogon application.   Let’s look at an example. Below, we use the sas-admin CLI’s audit plugin to see that user Ahmed logged in at 07:31 AM on the 25th of October (note that the timestamp is in UTC in PostgreSQL, but is converted to local time when a SAS Operations Infrastructure ETL job loads it to the AUDIT table in CAS).   /opt/sas/viya/home/bin/sas-admin --output text audit list --user-id Ahmed --sort-by ~timeStamp   ID Time Stamp Action State User ID Application URI c72c4533-886d-493b-9ef2-1dcff4ff4674 2019-10-25T07:57:54.776Z SessionDestroyed success Ahmed SASLogon bb748c54-6cb3-4edf-adbe-fce516c5786c 2019-10-25T07:31:21.014Z update success Ahmed reportImages /reportImages/jobs/4d7a64d8-1839-4d5b-a562-b140cdfc1f22/state caf1264d-4d37-444b-af45-4e339edc5dc6 2019-10-25T07:31:20.999Z create success Ahmed reportRenderer /reportRenderer/images 713e18ac-34db-4115-92aa-35f7662b0909 2019-10-25T07:31:20.649Z read success Ahmed reportData /reportData/commons/settings c5df6523-eb4f-42fe-b89b-fbdcc73d1c56 2019-10-25T07:31:20.298Z create success Ahmed reportPackages /reportPackages/jobs/b7325111-3f5a-4659-84bf-d1e59bdfbc74 e8578d9f-73b1-4dea-a94e-048a9eb9a72d 2019-10-25T07:31:20.030Z create success Ahmed reportData /reportData/jobs/61c55ce8-2b0a-4989-a14f-e1a5395b3b86 5aec043f-c2f8-46b6-a758-bc040abf1f69 2019-10-25T07:31:17.990Z create success Ahmed reportData /reportData/metadataSupplements fc7c64d8-bf75-40f5-a851-4afa5dc29beb 2019-10-25T07:31:16.141Z create success Ahmed reportImages /reportImages/jobs/4d7a64d8-1839-4d5b-a562-b140cdfc1f22 f9ad69f9-22b8-447b-82c2-45ebeb462409 2019-10-25T07:31:14.523Z read success Ahmed reportData /reportData/commons/settings 5dc96422-833c-496c-bd65-64659faf76d0 2019-10-25T07:31:08.641Z read failure Ahmed preferences /preferences/preferences/@currentUser/FormatLocale.DefaultLocale 5da5acf5-4d6b-47b3-9da5-c30ea7c3d946 2019-10-25T07:31:08.096Z read failure Ahmed preferences /preferences/preferences/@currentUser/_hc_xx_ 4a4c470f-afe9-469c-af3a-3fdb6d5f951e 2019-10-25T07:31:07.082Z read failure Ahmed preferences /preferences/preferences/@currentUser/FormatLocale.DefaultLocale b66f6eed-8ae8-47a3-8d3a-60e23df5cec9 2019-10-25T07:31:05.463Z login success Ahmed SASLogon   When a user logs off gracefully, a SessionDestroyed action is registered indicating the termination of the session as shown above.   If a user closes the browser instead of logging off, the SessionDestroyed does not get created in the audit records until the session times out. Session timeout duration is configurable.Set Time-out Interval for SAS Viya Web Applications in SAS® Viya® 3.5 Administration: Configuration ....   Some applications generate their own audit data when a user logs in. For example, SASStudio V creates a Compute server session when a user logs in, and an audit record similar to the following is captured.     /opt/sas/viya/home/bin/sas-admin audit list --user-id Ahmed --sort-by ~timeStamp --details --application compsrv   ID Time Stamp Type Action State Description User ID Application Remote Address URI 1967127d-87da-4656-922f-ed3b3bf1733d 2019-10-23T11:47:42.122Z resource start success 92ce6360-b23c-485b-94dc-48e5bd681bdd:https://intviya02.race.sas.com:38934/compute/servers/92ce6360-b23c-485b-94dc-48e5bd681bdd Ahmed compsrv   When the compute session is terminated, an audit record is captured indicating the deletion of the session (with the Description field indicating to the ID of the session).   /opt/sas/viya/home/bin/sas-admin audit list --user-id Ahmed --sort-by ~timeStamp --details --application compute   ID Time Stamp Type Action State Description User ID Application Remote Address URI 5ebd10ca-b4e1-4e7e-b71d-2874bfa672ca 2019-10-23T07:50:36.257Z resource delete success Deleted compute server 92ce6360-b23c-485b-94dc-48e5bd681bdd Ahmed compute /compute/servers/92ce6360-b23c-485b-94dc-48e5bd681bdd c9a1fc4a-48ea-4afa-a456-45e890571e42 2019-10-23T07:50:36.132Z resource delete success Deleted compute server session 92ce6360-b23c-485b-94dc-48e5bd681bdd-ses0000 Ahmed compute /compute/sessions/92ce6360-b23c-485b-94dc-48e5bd681bdd-ses0000   User sessions are assigned a session signature when they are established. The session signature (session_sig) property links the login event with the corresponding SessionDestroyed event. However, there are some important considerations.   First, refer to the caveats listed at the beginning of this post about potential issues with relying on the session signature for this purpose.   Second, the session signature is not visible when surfacing audit records using the CLI. This information is recorded in TSV files in the EV Datamart directory (/opt/sas/viya/config/var/lib/evmsvrops/evdm/application/security) on the Viya server. When a session is started:   2 34dca2cc-fcc9-4b6d-8c72-32d2958c9320 security application/vnd.sas.event.security 2019-10-15T06:21:18.973000+00:00 sasadm sas-deployment-id:dml5YQ==,sas-event-source:U0FTTG9nb24=,session_sig:NTNlZmNlZGE= security action:bG9naW4=,actionState:U1VDQ0VTUw==   Note the session_sig value. When the session ends, another record will be generated in the same file, referring to the original signature (orig_session_sig) for that session.   2 b41e8882-192d-4657-8fee-4cb04a96abda security application/vnd.sas.event.security 2019-10-15T07:02:30.282000+00:00 sasadm orig_session_sig:NTNlZmNlZGE=,sas-deployment-id:dml5YQ==,sas-event-source:U0FTTG9nb24=,session_sig:NDk0MGZmNw== security action:U2Vzc2lvbkRlc3Ryb3llZA==,actionState:U1VDQ0VTUw==   Fortunately, the legwork for extracting this information has already been done. Session information is extrapolated from these files and stored in the authentications subject table (opt/sas/viya/config/var/lib/evmsvrops/evdm/subjects/authentications.sas7bdat). Information is captured in this table for metering purposes, a secondary effect is that it provides a really nice way (with the caveats listed above!) to audit user session information. In the example below, several user sessions are listed, each with a datetime stamp of the login action, the session signature, and the duration of the session (in seconds).   Note that user Sasha’s session has a duration of 0.00 seconds. This points to an active user session (but there may be exceptions - listed above -  due to latency), indicating that Sasha has not yet logged out (SessionDestroyed action has not occurred for the session with this signature).   A job runs every 5 minutes to update this table, and another job runs daily to clear records older than 10 days. Update: To change the number of days (e.g. from 10 to 14 days) for which to capture records, execute:   /opt/sas/viya/home/bin/ops-dm-admin set EMI_DELETE_TSVZIP_DAYS=14   We can query this table to get some more insights about user sessions. Can you use it to figure out who was logged in at a certain time, for example?   A basic SAS program can be submitted to help answer that question.   /* Specify query time below in datetime. format */ %let qrytime='15OCT19:08:39:18'dt; libname evdm '/opt/sas/viya/config/var/lib/evmsvrops/evdm/subjects' access=readonly; data activesessions; set evdm.authentications; endtime=datetime+duration; format datetime datetime.; format endtime datetime.; where &qrytime between datetime and &endtime or (&qrytime >= datetime and (duration=0 and session_sig ne ''); run; proc print data=activesessions; run;   Note that this code sample is for demonstration purposes only, and there may be more efficient or more accurate methods to extract the required user session information.   Thanks to my colleagues Greg Smith, Carl Sommer, Fred Forst and Mike Roda for their contributions to this and the original version of this article, and thank you for reading! ... View more

PostgreSQL 15 is available in the November 2023 release of SAS Viya 3.5. SAS strongly recommends that customers upgrade their deployed PostgreSQL instance to version 15. Customers running SAS Viya 3.5 should check to see if they already have version 15 deployed, and if not, should consider upgrading in order to stay comfortably within PostgreSQL's support policy and to stay current with security fixes and patches. In this post, we will demonstrate how to check what version you are running in your environment, and highlight the steps you must perform to upgrade your deployed version of PostgreSQL to version 15.   If you are running a version of SAS Viya older than Viya 3.5, you must first upgrade your environment to SAS Viya 3.5 if you want to obtain PostgreSQL 15. Specifically, you should upgrade to at least Ship Event 23w44, which is the earliest release of SAS Viya 3.5 that contains PostgreSQL 15. This means that to get PostgreSQL 15, you must obtain a new order from SAS and then perform an upgrade. Any new deployments of SAS Viya 3.5 from 23w44 or later will deploy PostgreSQL version 15 by default. Similarly, if you are running a version of Viya 3.5 earlier than Ship Event 23w44, you need to upgrade to at least that release to get PostgreSQL 15. The simplest way to verify the version you are running is to directly query the version of PostgreSQL deployed in your environment by doing the following:   Determine the deployment target(s) for PostgreSQL in your environment In you inventory.ini file, look in the [sasdatasvrc] and [pgpoolc] (if applicable) host groups to identify the target machines, and then look in the [host_definitions] at the beginning of the file to view host information for those machines. Select any image to see a larger version. Mobile users: To view the images, select the "Full" version at the bottom of the page.   Check if the PostgreSQL 15 binaries have been deployed If SAS Viya is deployed on Linux, run the following command on the target machines: ls /opt/sas/viya/home/postgresql15 If not found, you do not yet have PostgreSQL 15 deployed.   Or if deployed on Windows, run the following in Powershell: dir "C:\Program Files\SAS\Viya\lib\postgresql15\bin"   If not found, you have not yet deployed PostgreSQL 15.   In these cases, you must contact SAS to obtain a new SAS Viya 3.5 software order from Ship Event 23w44 or later to upgrade to PostgreSQL 15. Once you have the new order, upgrade your SAS Viya deployment by following the documented instructions, remembering to perform the all-important pre-upgrade and post-upgrade steps for PostgreSQL. If you do have the PostgreSQL 15 binaries deployed, continue to the next step.   Verify whether your SAS Viya deployment is actually running the deployed PostgreSQL 15.   It is possible that your environment has PostgreSQL 15 deployed, but post-upgrade steps to configure it haven’t been performed. As an additional check, verify whether your SAS Viya deployment is actually running the deployed PostgreSQL 15.   On Linux, run the following as sudo on one of the target machines identified earlier: sudo cat /opt/sas/viya/config/data/sasdatasvrc/postgres/node0/PG_VERSION On Windows, run the following in Powershell:   cat C:\ProgramData\SAS\Viya\data\sasdatasvrc\postgres\node0\PG_VERSION A value of 15 listed in the PG_VERSION file indicates that PostgreSQL 15 is running and in use by SAS Viya. You can stop reading here, because all is as it should be!   However, if the listed version is not 15, some further action is required. We need to check to see if the latest PostgreSQL and related RPMs/MSIs are installed. On Linux, run the following to check the deployed RPM versions for PostgreSQL components:    cd /etc/init.d rpm -qa | egrep 'sasdata|sas-postgres|pool'   The versions displayed might resemble the below (the latter part of the values displayed for each RPM is irrelevant and has been omitted):   sas-sasdatasvrc-0.8.1-XXXXXXXXX.XXXXXXXX sas-postgresql-secure-libs-15-15.4-XXXXXXXXX.XXXXXXXX sas-pgpoolc-0.7.0-XXXXXXXXX.XXXXXXXX sas-postgresql-secure-15-15.4-XXXXXXXXX.XXXXXXXX sas-pgpool-II-44-4.4.4-XXXXXXXXX.XXXXXXXX If the versions shown above are displayed, this indicates you have deployed PostgreSQL 15 but not configured it yet. Some additional steps are required to reconfigure PostgreSQL and pgpool and complete the upgrade to version 15. Review the post-upgrade steps and completed all documented tasks.   If you do not have those versions, contact SAS to obtain a new SAS Viya software order from Ship Event 23w44 and upgrade your SAS Viya deployment in order to obtain PostgreSQL 15 Similarly, on Windows deployments, run the following in Powershell to check the MSI versions:    dir /powershell-deployment\Downloads   If sas-datasvrc15-15.4.0.0.msi   is listed, you have PostgreSQL15 deployed but not configured yet. Again, review the post-upgrade steps and completed all documented tasks. If you do not have that version, contact SAS to obtain a new SAS Viya software order from Ship Event 23w44 and upgrade your SAS Viya deployment in order to obtain PostgreSQL 15. A couple of additional points to consider:   Perform a full backup after updating your SAS Viya deployment After upgrading PostgreSQL, do not restore backups that were taken in older versions of PostgreSQL. If you have upgraded to PostgreSQL version 15, and you wish to then migrate to SAS Viya 4, you must move to a supported target version of SAS Viya 4 using an external instance of PostgreSQL 15.  Thank you for reading. My thanks also to my colleagues Cindy Taylor, Joseph Cho and Fred Perry for their contributions.        ... View more