Augmenting Streaming Events with Real-Time Rolling Statistics

[zephstemle]

Let’s say you would like to know if the current price of a stock is above or below the most up-to-the-moment average price, and by how much. Because the data is streaming, you would like to use SAS Event Stream Processor (ESP) to accomplish this but are not sure how it can be done.

This article and associated GitHub site provide all you need to construct an ESP model that meets our needs. The GitHub site includes a completed model, a starter model that you can finish, and test data. The site also includes step-by-step instructions to build your own model.

ESP Model

The ESP model would need to process the incoming stock prices, calculate the average, and then join the original price with its average. It would then calculate the difference between the current price and the average price. The model would output an event with the current price, average price, and the difference between the two. The model also needs to be fast and should perform at maximum efficiency.

Calculating Rolling Statistics

An Aggregate window is required to calculate the rolling statistics that will be joined with the original streaming event. This Aggregate window must be controlled so it does not become unbounded in growth. This is done by placing a Copy window before the Aggregate window that retains a finite number of events.

Join

To augment events with rolling statistics, we must configure a special type of join in the Join window. The streaming event from the Source window is the left side (fact) of a join and the aggregation value is the right side (dimension) of a left-outer join.

There are three requirements for the Join window to act in the way we need:

•   Tagged token event flow
•   Stateless joins
•   The No-regenerates option

Tagged Token Event Flow

Tagged token event flow enables each fork of a join to be tagged so that the side of the fork arriving first at the join waits for the tagged event of the other fork to arrive before performing the join and outputting an event. The Copy and Aggregate windows take longer to perform their operation than the event sent directly to the Join window. Therefore, the streaming event from the Source window waits for the corresponding event from the Aggregate window before outputting the augmented event.

Stateless Joins

Normally a Join window stores all events on the fact side of the join so it can go back and update any of them if a dimension changes. This makes the window stateful and can degrade performance over time. Because the join does not need to create a local copy of the left driving window (fact window) for updates, the join can be stateless.

No-regenerates Option

A typical left-outer or right-outer join re-generates corresponding fact events when the dimension side is changed. The type of join we want to perform should not re-generate fact events. The No-regenerates option optimizes memory usage by omitting the reference-counted copy of the fact window's index that is normally maintained in the Join window.

If the join type is left-outer or right-outer, the index is pi_EMPTY (stateless), and the No-regenerates option is turned on, this is as lightweight a join as possible.

For more information refer to the GitHub site for this project. There you will find all you need to build your own model.