SAS Event Stream Processing (ESP) software is the underlying engine which provides us with the ability to analyze and act on data in near real time as it flows into our system. Instead of waiting for the creation of an entire table of data to process in bulk, with ESP we can analyze each event almost as soon as it's generated. ESP is very flexible in supporting the ingestion of data from many different sources. Further, ESP can then pass the data from those events along to downstream systems or even create new events based on the results of analysis.
To manage the flow of data into ESP and then on out to other systems, ESP offers publish & subscribe (pub/sub) APIs which work in conjunction with ESP components referred to as Adapters and Connectors. Let's look specifically at these Adapater and Connectors and discuss how to decide which one is best for the task at hand.
ESP software is comprised of many parts. The entire system revolves around the ESP XML Server which is where the bulk of streaming analytics activities occurs. The ESP XML Server is made up of several parts itself, the heart of which is the Continuous Query engine which is in turn defined by the ESP Model. The ESP Model describes what the event data looks like with a Source Window and what to do with it with any Derived Windows as needed.
Data enters this ESP system through one or more Adapters and/or Connectors. And data can leave ESP bound for other ports downstream through one or more Adapters and/or Connectors as well. SAS provides a number of different Adapters and Connectors components in with the ESP software for our customers to use. Furthermore, if they're up for writing their own C/C++ code, ESP users can even create their own Adapters and Connectors. Or if they prefer to code in Java or Python instead, then they can create Adapters using those languages, too.
In ESP, Connectors are used to read from and write to event platforms. For example, ESP can read files directly from disk even as new lines are being written to them. Or skip the middle man and have the source process push those lines of information directly into ESP over a TCP socket. While direct connections like that can get the job done, they're not very robust in the face of network challenges. And so ESP provides several Connectors which communicate with message queue technologies instead, like IBM WebSphere MQ, Rabbit MQ, Solace, Kafka, and more.
Connectors are written in C++ and are loaded by the ESP XML Server when it's started so that they execute directly inside the ESP Engine.
In ESP, Adapters are also used to read from and write to event platforms - just like Connectors. As a matter of fact, Adapters run the same core code as their corresponding Connectors, just with additional wrapper to enable the Adapter to run as a standalone executable. This means that Adapters must be started up separately from the ESP XML Engine and further that they can run on a different host machine.
Pretty much every Connector has a corresponding Adapter - but there are several Adapters for which there is no equivalent Connector including several Adapters which are written in Java as well as Python. There are Adapter-only interfaces to SAS LASR Analytic Server, HDAT Reader, HDFS, Java Message Service, SAS Data Set, SAS CAS Analytic Server, and more.
So, in situations where you have a choice between using a Connector or an Adapter, which do you want to use? Like any sufficiently feature-rich software, the answer is: It depends.
To develop and maintain a simple environment with as few moving parts as possible, then Connectors are a good choice. Start up the main ESP XML Server and it automatically will load the available Connectors for use by your model. This is fantastic for learning how to work with ESP, delivering proof-of-concepts to customers, and for small-scale dev/test environments.
➥ EFFICIENT PERFORMANCE
All things being equal, the difference in performance between Adapters and Connectors in processing events is practically negligible. I don't want to oversell this concept; your mileage may vary especially where networking and your own coding skills come into play. But the takeaway here is that performance of event processing alone probably isn't enough to justify one over the other.
➥ RESOURCE CONTENTION
If the resources on your host computer are reaching full utilization, then that can limit the maximum amount of events which can be processed on the system. If you can offload some activities to other hosts, then that frees up those resources to process even more events. Since Adapters run independently as separate executables from the ESP XML Server, then they can be placed onto separate host machines, distributing the ESP workload across multiple machines.
➥ ENCRYPTED DATA TRANSFER
Transferring event data over the wire is increasingly subject to scrutiny these days. Even on purely internal networks with no routing over the public Internet, our customers want data transport encrypted from end-to-end. If your source system doesn't support transport-layer security (TLS, SSL) itself, then an ESP Adapter can be placed directly alongside and configured to encrypt data and feed it directly into the ESP engine. The same approach can be also be used for streaming event data downstream to other systems beyond ESP as well.
➥ EVENT COORDINATION
There are situations where you might need to feed events from two or more different sources into your ESP model. By default, events are treated asynchronously and so if you intend to perform some kind of join action, it's likely that the related events will pass through the model without finding each other. To help with this, ESP 3.1 and later offers Connector Orchestration which can be configured to coordinate how the model works with events as they come in.
Introduced in ESP 4.2, we also now have the Adapter Manager. It also helps to manage the coordination of events as they enter the ESP model similar to Connector Orchestration, but goes beyond that to offer event routing based on rules you define to implement with load balancing schemes as well.
ESP is the core technology software for the SAS IoT solution offerings. You owe it to yourself to get familiar with this wholly new approach to working with data. ESP empowers us to apply the power of SAS analytics to events as they're generated from a wide range of data sources.
Rob Collum is a Principal Technical Architect with the Global Architecture & Technology Enablement team in SAS Consulting. When he's not adapting his connectors and connecting his adapters, he enjoys sampling coffee and m&m’s from SAS campuses around the world.