SAS SpeedyStore Observability using SAS Enterprise Session Monitor
SAS Enterprise Session Monitor provides a wealth of information on a SAS Viya platform. It can also provide detailed information on a SAS SpeedyStore deployment. This post is a follow-on from my post on using the SingleStore Grafana dashboards, see: Using Grafana dashboards for monitoring SAS SpeedyStore
In this post, we will revisit the topic of monitoring, or what is often called Observability, but this time looking at using SAS Enterprise Session Monitor (ESM) to monitor a SAS SpeedyStore deployment.
As some context, SAS Enterprise Session Monitor isn’t designed to be a generalized reporting tool, like using Grafana dashboards. ESM is designed more as a Root Cause Analysis tool. It allows you to discover processes that are running in the SAS Viya platform and the Kubernetes nodes that are supporting the SAS Viya platform.
While it is not a tool for historical reporting, it does allow the export of data, which can then be reported on using SAS Visual Analytics. See the SAS Enterprise Session Monitor documentation: Viewing Reports with Extracted Data in SAS Visual Analytics
This function can be used to provide historical reporting. This could be to monitor infrastructure costs (it is possible to assign infrastructure costs in ESM), or to report on the users that are generating the most workload. For example, “Who are the top 10 SAS SpeedyStore users”.
You could say that SAS Enterprise Session Monitor will help you with the “How, What and “Why”.
For example, to answer questions like:
- What processes are running on the SingleStore nodes within my cluster?
- What processes are using all the resources?
- Why is my batch job running slowly?
Let’s look at using SAS Enterprise Session Monitor to monitor SAS SpeedyStore.
Observing the SAS SpeedyStore deployment
ESM provides a suite of functions that allow you to look at and/or filter by workload type. The Dashboard that you see on login has a ‘Load by Type’ portlet.
While this portlet will discover the SAS SpeedyStore processes by default, it is possible to configure the ESM Agent for additional workload types. To filter on specific pod types and define a custom name, the ‘esm-agents.yaml’ file has a set of regular expression statements that can be updated to meet your specific needs. The filters are defined under the ‘pod_types’ section.
For my testing I wanted to refine the filters for the ‘sas-singlestore’ pods. Using a text editor, I updated the ‘esm-agents.yaml’ file with the following settings:
pod_types:
- pod_log_level: WARN
pod_regex: .*-cluster-leaf.*
pod_type: S2-Leaf
- pod_log_level: WARN
pod_regex: ^node.*-cluster.*
pod_type: S2-Control
- pod_log_level: WARN
pod_regex: ^sas-singlestore-operator.*
pod_type: S2-Op
- pod_log_level: WARN
pod_regex: ^sas-singlestore-osconfig.*
pod_type: S2-daemon
When defining the rules, it is important to understand that they are processed sequentially, from top to bottom. So, you need to define the most specific rules first in the list.
This configuration allowed me to view the workload associated with the Leaf nodes separately from the Aggregator and Master nodes. The Aggregator and Master nodes have the type ‘S2-Control’ defined. At this point I should say that the default configuration would also allow this, it has the name S2C defined, but I wanted to update this name. I also wanted to add a definition for the operator and osconfig pods.
That was the only customization that I performed to monitor SAS SpeedyStore.
With the ESM Server and Agents in place, each of the Kubernetes nodes has an ESM Agent running on it. When a pod starts the ESM Agent “tags” the pod.
If you already had the ESM Agent deployed and then updated the agent configuration any new or changed “tags” won’t take effect until a new pod starts. If the change was aimed at one of the SAS Viya platform services you need to stop and restart SAS Viya. If the change was targeting the ‘sas-singlestore’ pods, you can pick up the change by pausing and restarting the SingleStore database (cluster).
For example, to pause the SingleStore cluster.
kubectl -n namespace patch MemsqlCluster sas-singlestore-cluster --type=json -p '[{"op": "add",
"path": "/spec/clusterState","value": "pause"}]'
Then, to restart the cluster.
kubectl -n namespace patch MemsqlCluster sas-singlestore-cluster --type=json -p '[{"op":
"path": "/spec/clusterState","value": "pause"}]'
With my updated configuration in place, I generated some workload using a batch job and using SAS Studio.
Using SAS Enterprise Session Monitor
In this post I won’t go into a lot of detail on using the ESM user interface, but I will pass on some hints and show a number of screenshots.
When you login to SAS Enterprise Session Monitor you will see the main dashboard, it shows a summary view of the Kubernetes cluster where SAS Viya is running. In this example, I had run my load test, so you can see some activity.
As part of the ESM Agent configuration you can set the cluster name, here I have set it to ‘SAS SpeedyStore’, as shown in the image.
Select any image to see a larger version.
Mobile users: To view the images, select the "Full" version at the bottom of the page.
Looking at the “Load by User” and “Load by Type” portlets, you can see the various workloads.
On the left, there is the “Load by User” portlet. As I was the only one using the platform it is a small list, it shows the ‘sasadm’ user that I used for running the batch workload. On the right you can see the “Load by Type” portlet. This shows my updated configuration using the S2-Control and S2-Leaf types. These names came from the ESM Agent configuration that I described above.
The third workload type to highlight is “Batch”, this is from my load test, as I used the SAS Viya CLI to submit a batch job. The two charts show a representation of the percentage workload.
Coming back to the first image, the two grey boxes shown in the “Jobs” portlet are my batch job (load-sashelp-tables). If you click on a job, it will open a “Timespans” report, this is shown in the image below.
The first UI tip is to use the ‘Configure Grouping’ function. I found the ‘group by Pod ID’ a good option. This means that the processes that you are viewing are grouped by the pod where they are running, and not just a list of processes. You will see this in the images.
Here the output was automatically filtered for me as I was viewing a single job. The “Process List” shows the SAS Batch Server pod name that is running the job.
This is where it is also important to understand the nature of the workload that you are viewing. For example, is it a batch job, a SAS Studio session using the SAS Compute Server, or a CAS session.
In this example, my load test was using the SAS Viya CLI to submit a batch job. Hence, it is using a SAS Batch Server. As stated above, you can see the pod name for this job on the left, in the Process List panel. You can also see the user that was running the job, the ‘sasadm’ user in this case. On the right you can see that we are viewing the breakdown for the ‘load-sashelp-tables’ job.
Let’s say I was trying to understand why my batch job was running so slowly. This is where ESM comes into its own.
The next screenshot is showing the compute node, the Kubernetes node:
‘aks-compute-90814698-vms000000’
This is where the batch job was running.
By using the mouse to hover over the chart you can see the details for the job. In the image above you can see valuable information on: IO (Device Reads and Writes), CPU and memory usage.
View statistics for a Node
You can also drill into specific nodes. This can be done in a few ways. The first option is to select a node from the main Dashboard. This is shown in the next image. Here you can see that using the “Nodes” portlet, I have selected the compute node (aks-compute-90814698-vms000000).
This provides a summary view for the compute node. The “Load by User” and “Load by Type” portlets are automatically filtered to show the details for the selected node. You can now see that the bulk of the workload was from the ‘sasadm’ user and was from the ‘Batch’ (SAS Batch Server) category.
The next example is doing a deep dive on one of the singlestore nodes in my AKS cluster. This shows the processes running on node ‘aks-singlestore-62860369-vmss000002’.
Using the category filters to just view the processes for the Leaf nodes (‘S2-Leaf’), you can see that the node has two leaf node pods running on it. On the bottom left you can see a process heatmap, and on the right you can see a view of the ‘IO Performance’.
What about tracking the interactive usage?
In my testing I also ran some workload from SAS Studio. This is illustrated in the following screenshots. The first is a Timespans view for the processes running on the Compute node.
Coming back to my comment on understanding the session flow to help focus your investigation.
When you start SAS Studio it spawns a SAS Compute Server session for you. This is why I needed to filter on Category ‘CMP’. This is shown in the tick-box on the bottom left of the image. You might notice that I also had ‘Batch’ selected. Here you can see one batch server session and two compute sessions in the Process List, based on this filtering.
Again, zooming in on the node, you can see the IO Performance for the associated ‘compute node’, the node where the SAS Studio session is running. It is also possible to get the details for an individual session, the session level IO (not shown here).
Looking at the first compute session in the Timespans view, you can see that this was for user ‘sastest1’. Looking at the details, you can see that I was running a PROC CASUTIL. This is shown in the top panel for the ‘sas.studio’ session.
One of the advantages of using SAS SpeedyStore is that when accessing a SingleStore database from CAS, using srctype='singlestore' on the CASLIB datasource options, the SAS Embedded Process is automatically invoked to support computed columns and WHERE clause processing.
ESM allowed me to get information on the SAS Embedded Process (EP) process (sasepmain) that is running on the Aggregator node.
In this example, you can see that I have selected the ‘sasepmain’ process and can now see the IO Performance associated with that process.
Again, you can see a breakdown of performance, this time for the Kubernetes node:
aks-singlestore-62860369-vmss000002
Finally, the image above also shows another example of using the “group by pod ID” for the Process List.
Conclusion
As you can see, SAS Enterprise Session Monitor can provide very in-depth information on the SAS Viya platform, including a SAS SpeedyStore deployment. Hopefully, this has given you a glimpse into the potential for using SAS Enterprise Session Monitor to monitor SAS SpeedyStore.
It was also very easy to set-up and deploy. I was up and running within minutes.
I wouldn’t throw away the Grafana dashboards that I describe in my previous post, as I feel they are complementary. With SAS Enterprise Session Monitor being more a diagnostic tool, and the Grafana dashboards are more focused on trend and historical reporting.
For example, in this last image, it shows the Grafana dashboard for disk usage (from when I was running my tests). For the testing I had 3 databases defined (MyDB, sales, and marketing). On the bottom left you can see the disk usage growing as I ran my tests.
The metrics database is the Grafana monitoring database discussed in the last post.
I hope this helps with your observability journey for monitoring SAS SpeedyStore.
Find more articles from SAS Global Enablement and Learning here.
