topic Authentication when using the URL_Connector to query data from a REST API in Streaming Analytics

Authentication when using the URL_Connector to query data from a REST API

VMoraal — Thu, 25 Jan 2024 14:16:18 GMT

Hi everyone,

I'm running into a little bit of an issue with the URL connector and using it to query data from an API.

To try and do a proof of concept, I had made a python script to connect and verified that the API spit out the data I expected in the correct format. Just like I did when hand-testing in the browser, I used the URL like so: https://id:pwd@somedomain.tld/api/folder/subfolder/object

This however seems to be causing some issues in SAS as the colon is used to indicate a port number and the @ is usually there to mark an environment variable. As a result, when trying to run my ESP project, I get the following error:

failed to retrieve data from https://id:0/api/folder/subfolder/object/, error Name or service not known

My next logical step for this was to try and remove the credentials from the URL to see if I could at least reach the API server. So I tried connecting to https://somedomain.tld/api/folder/subfolder/object/ and that did get me another error. After 300s of idling, I got a 403 response, indicating that the authentication service of the API must have timed out waiting for the login information.

Does anyone here have a tip on how I can manage to get this one API call to work within ESP?
Any tips and tricks are appreciated, I feel a little bit at the end of my wits here.

Re: Authentication when using the URL_Connector to query data from a REST API

AndyT_SAS — Wed, 31 Jan 2024 16:58:36 GMT

I don't think the url connector is expecting user/password information in the url. I will submit a request to update ESP to support that. What release of ESP are you using? That will help us find a work around.

Re: Authentication when using the URL_Connector to query data from a REST API

VMoraal — Thu, 01 Feb 2024 10:08:35 GMT

Hi AndyT,

Thanks for your answer 🙂 I'm currently on SAS Viya 2023.08 (v7.40.19), but...

Since posting my question, I have found that using a Basic HTTP authorization should theoretically work as I was able to successfully query the API with a curl command inside the pod where the ESP engine lives. While I couldn't find it in the documentation (I may not have been very thorough), The same Auth token can be used as an HTTP header. As a result, the previously mentioned login method is no longer necessary.

Maybe it would be an idea to include this sort of information in the documentation page for the URL connector?

My config file now looks like the one presented here:

<url-connector>
<url-publishers>
<url-publisher name='myconnector'>
<requests>
<request name="myconnector">
<url><![CDATA[https://api-url.tld/api/resource/]]></url>
<headers>
<header name="Authorization">Basic [insert-key-here]</header>
</headers>
</request>
</requests>
<generate>0</generate>
<opcode>upsert</opcode>
<event-loops>
<event-loop-json name='loop'>
<use-json>#_content</use-json>
<json>$.index</json>
<function-context>
<functions>
<function name='example'>json(#_context,'volume1.data')</function>
</functions>
</function-context>
</event-loop-json>
</event-loops>
</url-publisher>
</url-publishers>
</url-connector>

Hope someone with the same problem will have found this useful.