11-02-2017 11:41 AM
I have a problem with ESP flow implementation, which costs us a lot of RAM usage. As an input, I have a lot of raw records that are accumulating on daily, 7-days and 30-days level in realtime, and as a result, we have a huge amount of data. The doubt is whether to use stateful windows for these accumulated datasets or some persistent database (some in-memory DB). I would appreciate some best practices based on your experience with ESP, which methodology would suit the use case best? I am also considering that using persistent database between processing steps can help in failover, in case that the whole ESP is down.