topic Re: RTDM and JSON file integration in SAS Customer Intelligence

RTDM and JSON file integration

mysas1 — Fri, 12 Jan 2018 20:00:36 GMT

Hi Experts,

I am working with a client who has SAS RTDM. I need your guidance for following:

1. The Channel sends a JSON file which contains the request variables. However this file is hierarchical in structure which unfortunately SAS RTDM cannot understand. Hence I need to develop a code which will flatten this file and then give that flat file to SAS RTDM.

2. Again when the SAS RTDM sends the reply variables it is in flat JSON file which I need to convert to Hierarchical JSON file and then submit the file to the channel.

Could you guys help me understand is this the correct approach or if there is any other better and efficient way to do this.

Regards,

mySAS1

Re: RTDM and JSON file integration

ChrisHemedinger — Fri, 12 Jan 2018 20:03:32 GMT

Assuming you have access to a SAS programming environment, you can use the JSON libname engine to read the JSON as data and flatten to a SAS data set.

And you can use PROC JSON to generate a JSON file (flat or with hierarchies) from one or more SAS data sets.

Re: RTDM and JSON file integration

BillM_SAS — Fri, 12 Jan 2018 21:21:49 GMT

As Chris notes, the JSON procedure can create a hierarchical JSON file. I recently posted proof-of-concept code showing how to use PROC JSON to create a hierarchical JSON file from a data set in the SASHELP library. You can see that post via this link.

Re: RTDM and JSON file integration

Dmitry_Alergant — Mon, 15 Jan 2018 04:44:54 GMT

Hi,

Two responders above call for a programmatic approach using SAS language and its JSON handling capabilities.

They don't elaborate on how are you going to deploy the code so that it executed in real time as a web service, being a middleman between your client's system and SAS RTDM.

Probably, it is assumed that you will create SAS BI Web Service using SAS Stored Process as a backend, where you will your JSON manipulation code and publish it as a web service. I don't recall any other built-in way to deploy SAS code to respond web services in real-time. Except maybe through SAS Data Management Server (ex DataFlux) which your client is not likely to have.

SAS BI Web Services (Stored Process-based) approach should work, but also brings some robustness concerns. Depending on your deployment architecture, you may not currently have a fully redundant highly available set of components to run SAS BI Web Services / SAS Stored Process (and it takes more than just one component). I think these deployment challenges can be overcome, but it will need careful and thoughtful deployment architecture planning in advance.

If your client is more or less Cloud-Friendly and getting in-and-out of a corporate firewall is not that of a big deal, I would also consider a lightweight serverless programmatic solution using Amazon AWS Lambda function (in Node.js or Python) + API Gateways. It's a beautiful architecture, naturally robust, will only require a few lines of code and will be easy to maintain. It's built-in logging and monitoring metrics capabilities (AWS CloudWatch) can be further helpful in monitoring what happens with RTDM itself (by monitoring queries getting in and out of JSON gateway on its way).

Hope it helps. Good luck!

Re: RTDM and JSON file integration

Dmitry_Alergant — Mon, 15 Jan 2018 05:00:07 GMT

If you decide that SAS code-based approach (SAS BI Web Services & Stored Process) is not a fit, and the client is not ready to integrate through the cloud (AWS), then obviously a good old-fashioned traditional programmatic approach still applies. You develop a gateway web service using whatever programmatic platform you and the client like and then deploy it somewhere.

In our experience, with different our RTDM customers where it was required to put something "in between" of a calling system and RTDM (which happens for a lot of reasons), we have utilized a number of technologies:

A small custom Java web application deployed to SAS Web Application Server alongside RTDM engine itself
Some Python-based implementations
SAS Code / SAS Stored Process / SAS BI Web Services (as discussed in this thread)
AWS Lambda / API Gateways (as I mentioned in the previous post)
A service on Apache Mule ESB (free open source) that can be deployed alongside SAS RTDM
A service on other client-owned service bus solution (deployed on their ESB platform)

Obviously, all of that can be made to work reliably (plus a lot of other ways to implement a gateway web service) but requires varying effort to implement. A choice to be made may depend on customer's landscape, performance/HA requirements, customer's preferences, and available talent skill set.

Good luck!

Re: RTDM and JSON file integration

mysas1 — Mon, 15 Jan 2018 09:47:54 GMT

Hi All,

Thanks for your responses.

I reckon the BI service or the Lambda function, will eat in some execution time. Just wanted to check again since RTDM cannot understand hierarchical Jason, I will have to some how convert it into flat jason either through SAS programming or through Lambda. However the client doesnt have Jason engine hence I cannot use Jason libname and they are also not very keen on AWS Lambda. So two questions: Is there really a need to convert the Jason file from hierarchical to flat and then again flat to hierarchy and since I cannot use Jason libname or AWS Lambda what is the alternative for this here

Regards,

mySAS1

Re: RTDM and JSON file integration

Dmitry_Alergant — Mon, 15 Jan 2018 20:10:15 GMT

Hi, the JSON engine does not need any separate license, it's just baked in there in sufficiently recent SAS platform versions. If it's not there maybe SAS version is somewhat outdated and may be upgraded to a recent 9.4 maintenance release.

Even in this case, and if SAS BI Web Services is still your choice (despite high availability concerns), there are plenty of workaround approaches:

You can hard-code conversion logic using good old string manipulation functions without using specialized json libraries or engines
You can possibly export json to a physical file and then run whatever Linux command or script you like to operate on that JSON ("jq", python, or node.js), then load the results back. The execution delay may increase with all these back-and-forths, that's correct.
etc

Or you can implement a conversion web service using any other programming environment (not necessarily SAS BI or AWS Lambda) - Java, Python, Perl - whatever works for you and your customer. If implemented properly, an impact on execution delay may be negligible (like under 5ms easily).

At this moment I'm not sure what's your remaining question and how can we help you further. You may reach for more help on Stackoverflow...

If you question is "are there any controls in SAS RTDM that will somehow teach him to accept hierarchical JSON queries without external conversion, that it does not do currently based on your test" - then the answer is unfortunately no.

Good luck!

Re: RTDM and JSON file integration

mysas1 — Mon, 15 Jan 2018 22:59:42 GMT

Hi Dmitriy,

Thanks for your input. Helps me a lot, much appreciate

Regards,
mySAS1