01-24-2017 03:54 PM
Dear SAS Experts,
I need an advice for the best practice how to integrate IBM MQ client with SAS RTDM. Customer activities on the e-channels will be collected by Celebrus software and if event happens it will trigger Celebrus to send a message to the RTDM throught the IBM Websphere MQ service. On the back-end the message will end up in the IBM MQ client. What is the best practise to read these messages and transform them into REST/JSON calls to the RTDM ? And then to transform the response from RTDM into message for the MQ client? Many thanks.
01-25-2017 06:26 AM
01-26-2017 09:30 AM
thanks a lot for your comprehensive answer. I was actually thinking about possibility to mediate between IBM MQ and SAS RTDM using SAS interface to MQ and REST (through proc http). I thought if I create a windows service which indeed is a SAS session with program that contains an unlimited while loop which on one hand "listens" to the MQ client as a subscriber and calls REST API when the message appears in the queue. On another hand receives back responses and transfers them back to the queue as a publisher. But if I am right, according to your experience this would not be the efficient solution ?
Many thanks again.
01-27-2017 01:46 AM - edited 01-27-2017 01:53 AM
First, i'm not sure why you decided to go with the REST API way. An alternative is the SOAP API, which may be easier to work with (in my view). We use SOAP more often then REST. I think SOAP API was implemented in RTDM way earlier then REST, so it feels like more "native" and risk-free. But that's my just my personal opinion. Both APIs are supported and supposed to work.
Second, you'll need to decide on synchronous vs asynchronous query modes (which are supported by both REST and SOAP APIs).
A synchronous mode will be much easier to implement in your script, and i'm assuming this is what you meant. But it comes with a catch - with only one concurrent SAS loop script running, you will only process one request at a time in SAS RTDM. So unless the request workload is extremely low (and can't be the case for "customer activities in e-channels"), you will quickly start lagging behind with the growing queue and requests not being processed timely.
You can probably run a pool of several dozens SAS loop script sessions in parallel to overcome this problem, but it will soon become quite a complicated solution. You'll have to maintain and monitor a parallel session pool, restart failed sessions, power-cycle sessions after a certain amount of requests for the sake of hygiene, make sure your logs don't conflict, implement logs management and cleanup (there will be huge log volumes). If you want to support graceful shutdown or restart of a session that doesn't loose any event, you will have to develop some fancy logic to do that, as simply killing SAS process will likely catch it in between of processing and losing an event.
You will want to deploy all that on at least two SAS servers if you need high-availability, and you may not have compatible SAS license to do that!
Then I'd think of more advanced challenges. Depending on your use case and business logic requirements (but not always), you may need to maintain strict in-order processing of events that belong to the same customer e-channel session. This means not to begin new request processing until the previous request of same customer session was processed and returned a response. If this is the case, you won't be able to implement it with the looped SAS Base scripts approach.
Considering all above, I would recommend against using SAS scripts to mediate for real-time request processing in the marketing domain between e-channel and SAS RTDM. I suggest you implement this mediation logic using general purpose programming languages (like Java or Python), or an Enterprise Service Bus if you have one.
01-27-2017 06:16 PM
thanks a lot again for your comprehensive answer. I am sorry I got you a bit confused not actually writting the whole story. The whole story would be quite simillar but on the other hand a bit different. I was thinkig of a single windows service which is a SAS base code unlimited loop querying MQ client constantly for new messages. But whenever it encounters a message it actually asynchronously starts a new thread or SAS session with MP/Connect passing parameters/values to that session and leaving it to take care of the rest part - calling RTDM SOAP or REST APIs transferring a message back to MQ client as a publisher, creating an entry in the log file and terminating itself when finishes everything. The main loop can create as many concurent SAS threads or sessions as it needs. I think that the SAS session would be too slow to start therefore I was thinking about the threads. Do you think this approach could face similar challenges as you have described previously ?
Thanks a lot
01-30-2017 09:47 AM - edited 01-30-2017 09:48 AM
While your approach addresses some of the concerns that I mention, it creates a number of others. With your approach, you should be aware of the following risks and complications. Their severity depends on your real-time SLA requirements (maximum permissible delay for a single query processing) and workload (maximum number of requests per second).
I stand with my original point that while SAS scripts allow you to implement this integration (in principle), this is not an advisable solution to support any production workload short of extremely minimalistic.
Best regards, Dmitriy.