SAS Event Stream Processing (aka ESP) is an awesome product allowing to process and execute complex rules to hundreds of thousands (or even millions) of events per seconds.
There were many features and changes made in the ESP 4.x versions. So I thought it could be a good idea to introduce and summarize in a blog article some of the most exciting updates coming with ESP 4.x
If ESP was a TV show I would say: “If you enjoyed ESP season 3, I am sure you will love the new season !” : new heroes are stepping in (Cluster manager, Router, Generator), new fascinating horizons are explored (NiFi, Kafka, and Cloud), and a heart-stopping suspense on some very expected features (Streamviewer, MAS/DS2 integration, CAS integration etc...).
Today the production ESP version is 4.3, but this article will focus on the major changes since ESP version 3.2.
From the What’s new PDF that you can find in ESP 4.2 official documentation, the following topics are listed:
You can refer to this PDF for comprehensive descriptions of the new features.
In terms of documentation, be also aware that the examples are not provided as part of the ESP software but can be downloaded here from the SAS Support web site.
ESP 4.1 and 4.2 were only supported on RedHat Enterprise Linux is in the following version:
Although it is not reflected, at the moment, in the deployment guide, Oracle Linux (which is almost the same as RHEL and was used in CET testing) is also supported.
In addition, as with most of the others solutions that we have, the “SAS Support for Alternative System” policy applies. It allows a limited support (if you are running in to an issue using ESP 4.1/4.2 on a variant of the supported OS, then you might be asked to reproduce this very same issue on the officially supported OS). CentOS and others RHEL variants supporting RPM (RedHat Packaging Management system) also falls into this category.
With ESP 4.3, the support of Windows Operating system (64-bits) has been reintroduced (Windows 7, 2008 R2, 2012 R2).
In addition "ESP on the edge" (new with ESP 4.3) can now be deployed on the Edge (Network equipement, operational apps, gateways, devices, ...) so new OS such as Ubuntu and Yocto have added in the list of supported OS to allow ESP installation on ARM-64bit platforms. Follow this link to know more.
If the project implementation is driven, a discussion should take place to encourage the customer to choose a fully supported O/S as it will allow the customer to benefits from a complete support (both on Software and O/S side) during the project execution. It can be a key factor for the project success.
On the deployment side, ESP 4.x is now a YUM/RPM based installation. The Yellowdog Updater, Modified (yum) is an open-source command-line package-management utility for computers running the Linux operating system using the RPM Package Manager (source: wikipedia).
As with Viya, your order will include certificates allowing the customer server to access (over the public Internet network) to the SAS YUM repository.
The server will also need to be registered to a general YUM repository so that RPM dependencies can be pulled down if they are required during the install (RedHat/Centos should be registered to either an internal Satellite instance or the RHN, Oracle Linux would need to be registered to the ULN network).
In the case where customer’s machine cannot access the Internet ("dark sites") then alternatives are possible (use a local YUM repository).
Note that these requirements are not specific to ESP, they will also exist with any Viya deployment as it is using this new deployment process based on YUM/RPM.
One noticeable difference between ESP and Viya, though is that, for the moment, there are no "prepackaged" Ansible playbooks provided with ESP to automate system requirements configuration.
Finally, SAS Event Stream Processing is now cloud-ready. You can deploy it with tools such : Cloud Foundry BOSH (for PAAS: CloudFoundry, IAAS: Google Compute Engines, Apache CloudStack) or CHEF (Google Cloud Platform, OpenStack, Amazon EC2, MS Azure).
Streamviewer provides a web user interface that enables you to subscribe to window event streams (from one or more event stream processing engines), and display the events that stream through it.
You can display in real-time ("Dashboards that move") each event value as a row in a table or as an element of a graph.
Source : Steve Foerster's article
With the new Streamviewer:
Surely one of the most important change (at least on the Architecture side) is the ESP engines orchestration and “grid” publishing coming with the “Cluster Manager” : it allows elastic ESP architectures. As described below it is possible to distribute events from multiple edge devices to event stream processing engines that are provisioned within the Cloud.
The Cluster Manager automatically creates a "router" (which is another new ESP concept) in the XML Server to map events to corresponding engine instances. 3 event routing policies are supported:
The router uses the hash value to determine which engine instance the event is sent to.
The router enables you to integrate multiple event stream processing engines with one another:
It was already possible to do that with ESP 3.2, but this capability is now embedded in the engine and can work directly (bypassing the pub/sub API).
As illustrated below, the router configuration can be defined:
Note : in the first ESP 4.x the original name of this new feature was "Adapter Manager". Then it was renamed "Cluster Manager".
New connectors and adapters have been introduced:
Efforts were also done on the enhancements of the existing ones:
Apache NiFi is a core part of the Hortonworks Data Flow (HDF) offering and a great technology for collecting, processing, and moving data. Initially developed at the NSA, it supports secure, encrypted communication, bi-directional communication, data provenance to track lineage, and a configurable front-end with a variety of features.
SAS and Hortonworks worked to develop a NiFi processor that allows SAS machine learning models and advanced analytics to be embedded within the NiFi process flow.
A « .nar » file is provided for ESP and allows NiFi to use ESP processors ListenESP and PutESP for ESP integration.
A recent article highlights the benefits of such integration.
Event generator could be seen as a performance testing tool, it allows to generate event streams and publish into an event stream processing model.
SAS Micro Analytics Services (MAS) integration into SAS Event Stream Processing procedural windows to support Python and DS2 event stream input handlers.
IoT and Event Stream Processing are hot topics, today in the IT industry. SAS ESP is doing more than just catching up, it leads innovation (edge computing, in-stream analytics) and establishes its "Enterprise Class" software position (operational management, security) with these new features brought in each version since it entered in the 4.x versions range.
Hopefully this article gave you a good glimpse of the ESP new capabilities.