SAS ESP State Management Using In-Memory Databases

[jbhattacharya]

SAS Event Stream Processing (ESP) is engineered to process a high volume of events per second and provide low-latency response times on commodity hardware. The sub-millisecond latency is achieved in part by keeping the events in memory. With the increasing popularity of ESP in various use cases across industries, we are witnessing a requirement of retaining a huge volume of streaming and at-rest events for long periods. Some use cases for this are

•          Big Data static reference table to be used for lookup against streaming data.
•          Aggregation of streaming events over a huge time that runs into days or months.

In this video, we introduce the capability of ESP to integrate with an In-memory Database in the Kubernetes environment. With this feature, ESP can outsource the state management of Joins and Aggregations involving large retentions and lookups keeping the throughput and latency under acceptable limits.  The presentation highlights:

•          ESP integration with In-memory Database
•          ESP Kubernetes Operator framework support for the integration
•          Use cases where this integration can be leveraged
•          High-level architecture
•          Impact on the scaling of ESP server pods
•          Impact on failure recovery of ESP servers

As mentioned above, ESP leverages the ESP Kubernetes Operator Framework to achieve high throughput with minimum latency for integration with an in-memory database. For more details on the SAS ESP Kubernetes environment, you can visit the following Github entry: https://github.com/sassoftware/iot-esp-kubernetes-reference-architecture-guide

Be sure to check the video to know more about the feature and architecture of this brand-new functionality. Contact Divya Gupta or Joydeep Bhattacharya for questions and more information.