topic Re: About Apache NiFi and ESP integration in Streaming Analytics

About Apache NiFi and ESP integration

SundareshS — Fri, 06 Apr 2018 22:56:39 GMT

Hi all

Apologies that this query is more to do with a ESP related use case, rather than IoT. I am banking on the abundance of ESP knowledge to be found here.

SAS and Hortonworks have integrated SAS ESP with Apache NiFi (https://hortonworks.com/blog/moving-streaming-analytics-out-of-the-data-center/) , and I wanted to see if this will lend itself well to a "auto-suggest" / recommendation type capability on a software application.

Using some javascript/html services to capture an entered term, I can ingest this into an ESP window which then scores based on a model, but I don't want to write the result to some database / file. Rather, I want to direct it back to the same input form (much like the auto-complete features you see on a website)

Will the NiFi integration help in solving the last part of the problem - i.e. getting the results back ? Or, can we already accomplish this directly between the website and ESP?

regards,

Sundar

Re: About Apache NiFi and ESP integration

AndyT_SAS — Mon, 09 Apr 2018 19:02:12 GMT

Hi Sundar,

I'm not sure if Apache NiFi is the right tool or not. What piece of Apache NiFi would you use to return the suggestion to the browser? Would this be completely implemented on the browser side or would the browser make a request to a web server to get the scoring? If the web server is used, I'm wondering if the dfesp_rest_subscriber could be used to POST the webserver with the results of the scoring event in ESP. Then, that response could be given back to the browser. Another option, assuming ESP 5.1 is being used, would be to use the Websocket API to subscribe to the ESP model from the browser and receive the result back that way.

Thanks,

Andy

Re: About Apache NiFi and ESP integration

SundareshS — Thu, 12 Apr 2018 17:44:34 GMT

Hi Andy

Thanks very much for your reply. I was planning to call a NiFi process flow with the HandleHTTP request and HandleHTTPResponse at the start and end of the flow. This would be called from a web server, and within the flow, there will be :

- A PutESP processor which will publish the request to the source ESP window

- A ListenESP which will search for a response from the output / last window in the ESP model (this will be a ModelScore window)

However, from your answer, I get the impression we can accomplish the same objective even without NiFi, such as :

- Make a POST request from the web server to the ESP source window using a publish/subscribe API or a Web socket publisher (with the data along with an id value)

- Follow this up with another POST request from the web server to the ESP model through the dfesp_rest_subscriber (with the same id value)

The browser (or the search bar component in the browser) shall communicate with the server through an Ajax request, so that just the search bar container can be refreshed with the result.

Would this be possible?

regards

Sundar

Re: About Apache NiFi and ESP integration

AndyT_SAS — Mon, 16 Apr 2018 15:14:57 GMT

Yes, an Ajax request could be sent to the webserver, which could post an event to ESP, then process the response from the POST request received from the dfesp_rest_subscriber. This would eliminate the use of web sockets from the browser to ESP since all of the communication would be from the webserver to the ESP server. The browser wouldn't need to know any information about the ESP server.

Re: About Apache NiFi and ESP integration

SundareshS — Mon, 16 Apr 2018 15:28:36 GMT

Thanks very much. That makes things clear.

The only pending action would be to add some additional process for a fall back option - the two POST requests (for publish and subscribe) are asynchronous, therefore it is possible that there might not be an answer ready when dfesp_rest_subscriber is requested.

My feeling is that this may not be something that SAS can handle, one might need to write a program in JS/ web-server to fall back against a standard (less accurate) response - and return whichever response comes back faster - would that be correct?

regards

Sundar

Re: About Apache NiFi and ESP integration

AndyT_SAS — Mon, 16 Apr 2018 16:38:52 GMT

The dfesp_rest_subscriber adapter should always be running. This way, whenever an event arrives from the window being subscribed to, the POST request can be made to the HTTP endpoint. That would be better than having the adapter start/stop frequently.

I agree you will want a fallback mechanism to send a response if the POST request from the dfesp_rest_subscriber is not received in the required interval. This way the browser always receives a response to the initial request.