SAS ESP season 4: new characters and horizons

[#1]

SAS Event Stream Processing (aka ESP) is an awesome product allowing to process and execute complex rules to hundreds of thousands (or even millions) of events per seconds.

There were many features and changes made in the  ESP 4.x versions. So I thought it could be a good idea to introduce and summarize in a blog article some of the most exciting updates coming with ESP 4.x

If ESP was a TV show I would say: “If you enjoyed ESP season 3, I am sure you will love the new season !” : new heroes are stepping in (Cluster manager, Router, Generator), new fascinating horizons are explored (NiFi, Kafka, and Cloud), and a heart-stopping suspense on some very expected features (Streamviewer, MAS/DS2 integration, CAS integration etc...).

• Doc and examples
• Operating Systems and deployment
• A new Streamviewer
• Cluster Manager and Router
• New adapters and connectors
• Apache NiFi integration
• And it is not over...

Today the production ESP version is 4.3, but this article will focus on the major changes since ESP version 3.2.

Doc and examples

From the What’s new PDF that you can find in ESP 4.2 official documentation, the following topics are listed:



You can refer to this PDF for comprehensive descriptions of the new features.

In terms of documentation, be also aware that the examples are not provided as part of the ESP software but can be downloaded here from the SAS Support web site. 

Operating Systems & Deployment

ESP 4.1 and 4.2 were only supported on RedHat Enterprise Linux is in the following version:

• Red Hat Enterprise Linux 6.7 (64-bit) and later within 6.x
• Red Hat Enterprise Linux 7.1 and later within 7.x

Although it is not reflected, at the moment, in the deployment guide, Oracle Linux (which is almost the same as RHEL and was used in CET testing) is also supported.

In addition, as with most of the others solutions that we have, the “SAS Support for Alternative System” policy applies. It allows a limited support (if you are running in to an issue using ESP 4.1/4.2 on a variant of the supported OS, then you might be asked to reproduce this very same issue on the officially supported OS). CentOS and others RHEL variants supporting RPM (RedHat Packaging Management system) also falls into this category.

With ESP 4.3, the support of Windows Operating system (64-bits) has been reintroduced (Windows 7, 2008 R2, 2012 R2).

In addition "ESP on the edge" (new with ESP 4.3) can now be deployed on the Edge (Network equipement, operational apps, gateways, devices, ...) so new OS such as Ubuntu and Yocto have added in the list of supported OS to allow ESP installation on ARM-64bit platforms. Follow this link to know more.  

If the project implementation is driven, a discussion should take place to encourage the customer to choose a fully supported O/S as it will allow the customer to benefits from a complete support (both on Software and O/S side) during the project execution. It can be a key factor for the project success.

On the deployment side, ESP 4.x is now a YUM/RPM based installation. The Yellowdog Updater, Modified (yum) is an open-source command-line package-management utility for computers running the Linux operating system using the RPM Package Manager (source: wikipedia).

As with Viya, your order will include certificates allowing the customer server to access (over the public Internet network) to the SAS YUM repository.

The server will also need to be registered to a general YUM repository so that RPM dependencies can be pulled down if they are required during the install (RedHat/Centos should be registered to either an internal Satellite instance or the RHN, Oracle Linux would need to be registered to the ULN network).

In the case where customer’s machine cannot access the Internet ("dark sites") then alternatives are possible (use a local YUM repository).

Note that these requirements are not specific to ESP, they will also exist with any Viya deployment as it is using this new deployment process based on YUM/RPM.
One noticeable difference between ESP and Viya, though is that, for the moment, there are no "prepackaged" Ansible playbooks provided with ESP to automate system requirements configuration.

Finally, SAS Event Stream Processing is now cloud-ready. You can deploy it with tools such : Cloud Foundry BOSH (for PAAS: CloudFoundry, IAAS: Google Compute Engines, Apache CloudStack) or CHEF (Google Cloud Platform, OpenStack, Amazon EC2, MS Azure).

A new Streamviewer

Streamviewer provides a web user interface that enables you to subscribe to window event streams (from one or more event stream processing engines), and display the events that stream through it.

You can display in real-time ("Dashboards that move") each event value as a row in a table or as an element of a graph.

Source : Steve Foerster's article 

With the new Streamviewer:

• We can now save and load subscriptions and SAS graphs and their customized settings in dashboards. Dashboards are saved in a configuration database.
• Graphs and tables have been visually enhanced and we are not relying anymore on Google charts, so Internet (or at least the google web pages) access is not required anymore.

Note that with ESP 4.3, there are now two ways to embed Streamviewer components in external web pages Javascript and Iframes.

Cluster Manager and Router

Surely one of the most important change (at least on the Architecture side) is the ESP engines orchestration and “grid” publishing coming with the “Cluster Manager” : it allows elastic ESP architectures. As described below it is possible to distribute events from multiple edge devices to event stream processing engines that are provisioned within the Cloud.



The Cluster Manager automatically creates a "router" (which is another new ESP concept) in the XML Server to map events to corresponding engine instances. 3 event routing policies are supported:

• Multicast policy : sends every event to all the engine instances
• Round Robin policy : sends events to engine instances in a round robin fashion
• Hash policy : hashes the value of some pre-defined field(s) and uses that value to decide where to send the events: the hash value is an integer between 0 and the Number of engine instances minus one.

The router uses the hash value to determine which engine instance the event is sent to.

The router enables you to integrate multiple event stream processing engines with one another:

• One engine can do data preparation and dimensional lookups. Another engine can subscribe to the first and perform pattern matching.
• Events can be routed by examining the contents of events (fields) and then send events to one of many downstream engines.

It was already possible to do that with ESP 3.2, but this capability is now embedded in the engine and can work directly (bypassing the pub/sub API).

As illustrated below, the router configuration can be defined:

• Either in the standard XML server model configuration directly beneath the engine element: events are directly injected into projects, bypassing the pub/sub API.
• Or the router configuration can be the model itself: This runs an XML server that acts solely as a router (using the publish/subscribe API between engines.)

   

Note : in the first ESP 4.x the original name of this new feature was "Adapter Manager". Then it was renamed "Cluster Manager".

New adapters and connectors

New connectors and adapters have been introduced:

• Kafka: the new Apache pub/sub system replacing traditional Message brokers as RabbitMQ. Kafka is also one of the supported Messaging system supported for ESP failover architectures.
• MQTT: Lightweight Pub/Sub messaging protocol (MQ Telemetry Transport). Designed to minimize network bandwidth and device resource requirements (suitable for M2M, IoT and mobiles apps).
• Apache Cassandra (NoSQL distributed DB), Twitter GNIP, BoardReader, CAS

Efforts were also done on the enhancements of the existing ones:

• ArcSight CEF format support, Sniffer TLS initial connection, encrypted passwords
• Additional options and capabilities for connectors and adapters (opcode and flags in CSV, quiesce parameter, etc...)
• New pub/sub API : you can now develop your own ESP pub/sub APIs with Python 2.x or 3.x (in addition of C and Java)

Apache NiFi integration

Apache NiFi is a core part of the Hortonworks Data Flow (HDF) offering and a great technology for collecting, processing, and moving data. Initially developed at the NSA, it supports secure, encrypted communication, bi-directional communication, data provenance to track lineage, and a configurable front-end with a variety of features.

SAS and Hortonworks worked to develop a NiFi processor that allows SAS machine learning models and advanced analytics to be embedded within the NiFi process flow.

A « .nar » file is provided for ESP and allows NiFi to use ESP processors ListenESP and PutESP for ESP integration.

A recent article highlights the benefits of such integration.


 

And it is not over...

Event generator could be seen as a performance testing tool, it allows to generate event streams and publish into an event stream processing model.

• A generator definition must be defined in an XML document.
• Then the event generator HTTP interface is used to send the Generator definition and specify the number of events to inject, maximum rate per second, etc...

SAS Micro Analytics Services (MAS) integration into SAS Event Stream Processing procedural windows to support Python and DS2 event stream input handlers.  

Spoilers (ESP 4.3...)

• ESP 4.3 is GA release only since teh last month (week 21), here are some of the key changes in this version :
  • Advanced Analytics : New windows : Score, Train, Calculate, Model Reader, Supervisor, New capabilities : ASTORE support, Out of stream and In stream.
  • Edge computing : support of ARM 64-bit Linux
  • Enhanced integration with the Viya In-Memory Analytics engine, CAS (Cloud Analytics Services) 
  • Operational Management : Event Stream Manager to deploy and monitor ESP models
  • New ESP Studio (integrated XML editor)
  • New Security capabilities : SAS Logon authentication, Kerberos, User Access Control, ...
    
    

Conclusion

IoT and Event Stream Processing are hot topics, today in the IT industry. SAS ESP is doing more than just catching up, it leads innovation (edge computing, in-stream analytics) and establishes its "Enterprise Class" software position (operational management, security) with these new features brought in each version since it entered in the 4.x versions range.
Hopefully this article gave you a good glimpse of the ESP new capabilities.