Real time data monitoring using SAS DataFlux DM studio

[Che]

Hello,

Has anyone implemented real time data monitoring using SAS DataFlux Data Management Studio?

I have build some Data monitoring jobs that runs in batch, looking for some guidance on how real time monitoring can be set up using MD studio.

Best regards,

Chetan

[RonAgresta]

Hi Chetan - 

Assuming that by "real-time monitoring" you mean capturing data metrics or violations on a per-record basis through a web service, you have already done most of the work. By creating a variation of your batch process that uses an External Data Provider node instead of a data source, you will have created a job that can be deployed as a web service. Upload that new job to Data Management Server and then you can call that job as a web service (SOAP or REST based, your choice).

If you want to differentiate the results that end up in your data monitoring repository as real-time versus batch sources, you can add new data elements as properties to the business rules (and selectable in the Data Monitoring node) to call out the systems, owners, priorities, or statuses associated with the system that made the web service call. These can be parameterized too such that you can pass the parameters into the web service from the web-service invoking system when the call is made.

Ron

[Che]

Hello Ron,

First thanks for your time and response to my post. By real time, I mean if any new record is added to database that meets business rule defined, I would like to send an email or write that record to a separate table.So business users are notified quickly and they will take action asap.

[RonAgresta]

Your business rules in combination with the Data Monitoring node can take care of this. Review this page in the Data Management Studio User's Guide. When a business rule is triggered, you can use events to write to tables and send emails.

[Che]

Thanks Ron.

I will try to implement this and get back to you if I have any questions.

Best regards,

Chetan

[Che]

Hello Ron,

Here is what I am trying to accomplish:

I have a table with few million records which gets updated in real time. I have to monitor value of one column and if I find a record with that value, I need to report it or write it to a table with create date and derive a real time dashboard.

What I have Done so far:

I set up fields, tasks and business rule using SAS Data Management Studio.

I have set up data job which read from table and uses monito node which in result triggers a event which essentially write that record to a table.

My SQL currently checks for any new records that were added in the last five minutes, so reporting is always on new records added in last five minutes.

Question:

For real time monitoring, do I need to run this job every five minute or is there a way to inject this at database level?

Regarding dashboard that I talked about, do you recommend on using DM Studio for this or I should look for some other tool?

Any inputs/help here is greatly appreciated.

Best regards,

Chetan

[RonAgresta]

You would either have to run this job periodically using some change data capture technique to only process data you have not seen or you could create that real-time service I mentioned and see if you can make a web service call from your database (which is not trivial). Where you present your results is up to you. SAS Data Management Studio has a built in dashboard but it is desktop based. If you want to share this information with a broader audience, you could use SAS Visual Analytics or another reporting environment. But in either case for web-based reporting you will need to write database queries to pull data out of the DataFlux repository for consumption in your reporting environment.