Sorting data in SAS ESP

[upadhi]

Hello,

We want to group data according to an id being processed in the streaming data. Is there any way we can group data in SAS ESP? and Is there a way where we can pause data processing until all the parallel nodes are done processing one record, so that new record is taken only after all the parallel nodes have processed one record.

Thanks in advance.

[ScottK · Accepted solution]

The second part of the question may be easier, so let's discuss that first. If you have a single source window where a event enters the ESP model, and splits out to parallel compute paths and is rejoined later on, you have what is called a fork-join. SAS ESP supports tagged token data flow which will do exactly what you want in this case, allow all forks to complete and a single result produced by the model where all the paths rejoin. In xml this is enabled by specifying the use-tagged-token='true' attribute on the project element. So you get what you want  without pausing the processing at all.

The sorting question is more difficult in general for any streaming engine. Events that are produced by a streaming engine are normally supplied to a consumer as soon as the streaming engine finishes its computation. This means the streaming engine does not hold a bunch of events, and then a consumer issues a query against the window and gets a result set back. That is more of a database paradigm. If you want to talk through the use case you have that requires sorting data, I would be happy to work with you and see how it might be architected.  

[upadhi]

Hey  Scott,

Thank you so much for the help. It worked exactly the way we wanted.!!