topic JSON response from URL Connector can not be parsed in Streaming Analytics

JSON response from URL Connector can not be parsed

VMoraal — Fri, 02 Feb 2024 10:18:44 GMT

Hi Everyone,

I am looking for advice on a problematic JSON response from an API.

The API response in question contains a "formula" field which contains some kind of interval written as: "x>400 && x<1700".

From what I can see in other tools, including Python, Lua, Notepad++ & Firefox, the JSON is valid and parses correctly. In ESP however, when the JSON comes through, no error shows, but no data comes through either.

When looking at the ESP-Server-Pod's logs I saw that that "formula" field was throwing errors because of unexpected or wrong html syntax... (see log screencap attached)

Does anyone have any idea how I can force ESP to just pass the response to a proper JSON parser and just ignore these HTML errors?
Because the way I see it, these errors aren't relevant for the parsing of the JSON as #1: this field is ignored, and #2: this isn't HTML it's JSON...

Thanks in advance for the help and advice 🙂

Re: JSON response from URL Connector can not be parsed

VMoraal — Fri, 02 Feb 2024 10:24:05 GMT

Additional context to the question above.

The API in question is being queried using the URL Connector with an XML file which defines the following headers as well as the JSON parsing for events generation:

Authorization: "Basic [key]"

Content-Type: "application/json"

Re: JSON response from URL Connector can not be parsed

VMoraal — Fri, 02 Feb 2024 12:56:25 GMT

Dear VMoraal,

I think it is time for you to pack your bag and start your weekend... You clearly can't read.

The problem was due to the lack of declaration of the content-type in the request name. Somehow my brain was stuck somewhere else.

Other users beware! If you don't declare the content type, ESP will read the response of your request as HTML.

<request name="api_query" contentType="application/json">