SAS Event Stream Processing (ESP) Reference Architectures

[bkooman]

In this GitHub project, we will share different reference architectures of ESP and how one can evaluate which one out of these is the best fit for your customer/partner use-case. Figure 1 below provides the steps that one must follow to determine the right architecture for their business problem.

NOTE: In this project we do not provide any guidelines on the ESP project modeling.

Figure 1: Steps to Determine the Reference Architecture

 

2. Determine the Use Case Requirements

To determine the right architecture, it is important to understand the customers' long-term business objectives. The below sections explains

1. The different types of design requirements to consider and their importance,
2. Questionnaire helps to ask the right questions to determine the right requirements, and
3. Following the flowchart/Decision-Tree to reach the architecture that fulfills all the business goals.

 

Understanding the ESP Architecture Design Requirements

Figure 2 presents the non-functional, functional, infrastructure, and streaming input/output requirements for a use-case to determine the right ESP architecture.

Non-Functional requirements define the quality attributes of the applications, i.e., ESP. These requirements allow the architect to impose constraints or restrictions on the design of the overall architecture.

Functional requirements define a system and its components that are required by the application to perform the desired behavior. They primarily are derived from the use-case objectives, what will the input data sources and outputs, what should be the format of the data in consideration, how the application must process it, and finally, verifying the correctness of the obtained output. It is usually defined in terms of calculation, transformations, data manipulation, business processes, user interaction, and various tasks to be performed by the ESP.

By Infrastructure requirement we mean the platform to be used for deploying the ESP. Using Cloud adds several benefits as the cloud-native services can be leveraged and many 3rd party tools/applications in need can be managed services provided by the Cloud platforms. This comes at a cost.

We will also look into Streaming Input/Output which means how from where and what format of the input data/events are streamed to the ESP and where the output results must be streamed to. We also consider the volume, velocity, and variety of the streaming events as well.

In this GitHub project, we will focus on the non-functional and streaming input/output design requirements to determine the right infrastructure and select the right architecture.

Figure 2: ESP Architecture Design Requirements

 

Definitions of Non-Functional Requirements

The ESP reference architectures use the standard definitions from Cloud Computing (Distributed Systems).

 

Real-time Streaming Architecture Design Guide Questionnaire

During designing or architecting a solution, we must ask questions! A lot of questions! The Streaming Architecture Design Guide Questionaire provides a comprehensive list of questions one must ask to architect a complete solution based on the provided business requirements.

 

3. Decision-Tree to Determine the Reference Architecture Based on the Requirements

Figure 3 presents the Design-Tree/Flowchart which is built using the possible business requirements a customer/partner would have. The flowchart helps in determining the right ESP reference architecture that fulfils the business needs. It is also possible to have mix of many architectures, if many paths in the flowchart are realized. It is extremely necessary to have the complete list of requirements to obtain the most suitable architecture.

Figure 3: Design Tree to Determine the Reference Architecture

 

4. Reference Architectures

• 01. Single-Node, Single ESP Server Deployment
• 02. ESP with traditional High Availability with and without Kubernetes
• 03. Multi-Node, ESP Operator Kubernetes Framework
• 04. Multi-Node, ESP with Kubernetes using Kafka
• 05. Multi-Node, ESP with Kubernetes using Load Balancer