We’re smarter together. Learn from this collection of community knowledge and add your expertise.

SAS ESP Event Generator

by SAS Employee NicolasRobert on ‎05-12-2017 09:06 AM (996 Views)

SAS Event Stream Processing is a powerful solution to analyze streaming data and take appropriate action instantly. When designing your ESP model, you might face some challenges regarding the test, simulation, performance analysis and workload management of your streaming application. That’s where the Event Generator comes in. An Event Generator is an ESP object that generates event streams and publishes them into an event stream processing model. The Event Generator, which is new in SAS ESP 4.1/4.2, is very useful:

  • To generate random sample data
  • To test your ESP model when you don’t have yet sample data in flat files or in dynamic sources
  • To validate the performance, the throughput, the low-latency required and the architecture of your ESP application

Let's go into more details.

 

Design an Event Generator

 

An Event Generator is an XML string that cannot be designed in ESP Studio. Event Generators can be created, destroyed, started, and stopped through the SAS Event Stream Processing HTTP interface (REST admin API). Let’s have a look at an Event Generator complete example:

 

<event-generator insert-only='true' autogen-key='false'>
 <publish-target>dfESP://<server>:5555/tradesEventGenerator/trades/TradesSource</publish-target>
 <resources>
  <list name='symbols' delimiter=' '>SAP IBM XOM LQD LPX AOL TPC</list>
  <list name='brokers' delimiter=' '>10 11 12 13 14 15</list>
  <list name='venues' delimiter=' '>1 2 3 4 5</list>
 </resources>
 <init>
  <value name='ID'>i64(0)</value>
  <value name='udate'>i64(timeParse('08/01/2016 00:00:00','%m/%d/%Y %H:%M:%S'))</value>
  <value name='numBrokers'>listSize(#brokers)</value>
  <value name='numSymbols'>listSize(#symbols)</value>
  <value name='numVenues'>listSize(#venues)</value>
 </init>
 <fields>
  <field name='ID'>i64(sum($ID,1))</field>
  <field name='symbol'>listItem(#symbols,random(0,$numSymbols))</field>
  <field name='currency'>87236</field>
  <field name='udate'>i64(sum($udate,random(0,3)))</field>
  <field name='msecs'>random(1,10000)</field>
  <field name='price'>random(1.5,200.0)</field>
  <field name='quantity'>random(1000,5000)</field>
  <field name='venue'>listItem(#venues,random(0,$numVenues))</field>
  <field name='broker'>listItem(#brokers,random(0,$numBrokers))</field>
  <field name='buyer'>random(1,1000)</field>
  <field name='seller'>random(1,1000)</field>
  <field name='buysellflg'>random(0,1)</field>
 </fields>
</event-generator>

 

The "publish-target" tag identifies the ESP source window that we want to publish events to.

 

 <publish-target>dfESP://<server>:5555/tradesEventGenerator/trades/TradesSource</publish-target>

 

The "resources" tag defines the resources from which you can read maps, lists, and sets to be accessed from your event generation functions.

 

 <resources>
  <list name='symbols' delimiter=' '>SAP IBM XOM LQD LPX AOL TPC</list>
  <list name='brokers' delimiter=' '>10 11 12 13 14 15</list>
  <list name='venues' delimiter=' '>1 2 3 4 5</list>
 </resources>

 

Resources can also point to external files.

 

 <resources>
  <list-url name='symbols'>http://<server>:18080/esp/stuff/tradesData/symbols.txt</list-url>
  <list-url name='brokers'>http://<server>:18080/esp/stuff/tradesData/brokers.txt</list-url>
  <list-url name='venues'>http://<server>:18080/esp/stuff/tradesData/venues.txt</list-url>
 </resources>

 

The "init" section is aimed to initialize a set of values upon the start of the event generation. The resources (properties) defined above are referenced here using # symbols (e.g. #brokers, #venues).

 

 <init>
  <value name='ID'>i64(0)</value>
  <value name='udate'>i64(timeParse('08/01/2016 00:00:00','%m/%d/%Y %H:%M:%S'))</value>
  <value name='numBrokers'>listSize(#brokers)</value>
  <value name='numSymbols'>listSize(#symbols)</value>
  <value name='numVenues'>listSize(#venues)</value>
 </init>

 

And the "fields"tag defines the event values to be generated using a set of functions. They are based on the functions defined in Functional Window and Notification Window Support Functions in SAS Event Stream Processing: Programming Reference.

 

 <fields>
  <field name='ID'>i64(sum($ID,1))</field>
  <field name='symbol'>listItem(#symbols,random(0,$numSymbols))</field>
  <field name='currency'>87236</field>
  <field name='udate'>i64(sum($udate,random(0,3)))</field>
  <field name='msecs'>random(1,10000)</field>
  <field name='price'>random(1.5,200.0)</field>
  <field name='quantity'>random(1000,5000)</field>
  <field name='venue'>listItem(#venues,random(0,$numVenues))</field>
  <field name='broker'>listItem(#brokers,random(0,$numBrokers))</field>
  <field name='buyer'>random(1,1000)</field>
  <field name='seller'>random(1,1000)</field>
  <field name='buysellflg'>random(0,1)</field>
 </fields>

 

For example, the "listItem(#symbols,random(0,$numSymbols))" will return a random item from the defined list of symbols, the random function returning a number between 0 and $numSymbols.

 

Run an Event Generator

 

Once you have designed a valid Event Generator, next steps are to create it by pushing the XML into an ESP XML server and start it. Here is the command to create it. The URL means that you want to create an Event Generator called myFirstEG in the according ESP server.

 

dfesp_xml_client -url "http://<server>:5556/SASESP/eventGenerators/myFirstEG" -put "file://tradesEG.xml"

Note: this example assumes you have an ESP XML Factory Server running with an admin port of 5556.

 

You should see the following:

 

nir_1_eg_creation.png

 

If you look to the linux terminal where you started the ESP XML Factory Server (assuming you started it with the -loglevel "event.generator=trace"), you should see how the Event Generator is initialized:

 

nir_2_creation_console.png

 

Then, you are ready to start your Event Generator with the following command:

 

dfesp_xml_client -url "http://<server>:5556/SASESP/eventGenerators/myFirstEG/state?value=started&events=3000&blocksize=5&rate=50" –put

 

This command tells the Event Generator to load 3000 events, at a rate of 50 events per second with a blocksize of 5 events.

 

You should see the following:

 

nir_3_eg_start.png

 

If you look to the linux terminal where you started the ESP XML Factory Server (assuming you started it with the -loglevel "event.generator=trace"), you should see how the Event Generator publishes events:

 

nir_4_eg_start_console.png

 

Now it is running, you should see some events coming in your ESP model, so that you can check the consistency of your model and your rules.

 

You have additional REST API methods:

Finally, the real rate (number of events published per second) is defined by the maximum value of either blocksize or rate. Here are several examples to illustrate this:

  • EVENTS=100 BLOCKSIZE=10 RATE=1
    • 1 block of 10 events will be published per second, resulting in a 10 seconds run
  • EVENTS=100 BLOCKSIZE=1 RATE=10
    • 10 blocks of 1 event will be published per second, resulting in a 10 seconds run
  • EVENTS=16384 BLOCKSIZE=256 RATE=1024
    • 4 blocks of 256 events will be published per second, resulting in a 16 seconds run
Comments
by New Contributor Rain
on ‎06-01-2017 08:46 AM

Thank you for providing this good explanation how to use Event Generator functionality. I was trying to use list-url as a resource but ran into small problems.
The resource was described as follows:
<list-url name='symbols'>http://xxxxxx:8090/static/symbols.txt</list-url>

 

Initialization contained this line:
<value name='numSymbols'>listSize(#symbols)</value>

 

Symbols.txt contained these:
SAP
IBM
ORA
MSF
OMX

 

Now, when I run generator it creates some empty symbol values and when I used additional field to monitor  the size of numSymbols it gives me 10. I guess the reason for that is function listSize cannot be applied to resource type url-list? Any suggestions how I could solve this to get random elements from Symbols.txt file?

 

Thank you.

Rain

Contributors
Your turn
Sign In!

Want to write an article? Sign in with your profile.