topic Re: Streaming analytics confusion in Streaming Analytics

Streaming analytics confusion

okomo — Fri, 22 Mar 2024 13:27:49 GMT

Hey chaps.

I’m back playing with streaming analytics this time. We’ve got many IoT sensors providing data at various points throughout the day. Some report back every 30 seconds, others 3 minutes, others twice a day, some even once a day so it all varies.

In total, we get around 1.5 million data points a day.

I’ve got a task to stream this data to

a real-time power bi dashboard

expose the data to sql to allow it to be consumed for further analytical work.

Question:

I don’t feel very confident using this stack, I changed the sql database schema to generate an error. I didn’t get told the output was failing.

I knew it was so went digging, but in reality I’d like an alert or something to shout at me. Is this possible?

The streaming analytics job is configured to not drop so it’ll be ‘stuck’.

If I configured it to drop, I’d there a way of knowing what was dropped?

Re: Streaming analytics confusion

ballardw — Fri, 22 Mar 2024 15:31:24 GMT

Exactly where does SAS software come into this process?

Is the SAS portion doing what it is supposed to?

@okomo wrote:

Hey chaps.

I’m back playing with streaming analytics this time. We’ve got many IoT sensors providing data at various points throughout the day. Some report back every 30 seconds, others 3 minutes, others twice a day, some even once a day so it all varies.

In total, we get around 1.5 million data points a day.

I’ve got a task to stream this data to

a real-time power bi dashboard

expose the data to sql to allow it to be consumed for further analytical work.

Question:

I don’t feel very confident using this stack, I changed the sql database schema to generate an error. I didn’t get told the output was failing.

I knew it was so went digging, but in reality I’d like an alert or something to shout at me. Is this possible?

The streaming analytics job is configured to not drop so it’ll be ‘stuck’.

If I configured it to drop, I’d there a way of knowing what was dropped?

Re: Streaming analytics confusion

AnnetteDio — Fri, 22 Mar 2024 16:06:00 GMT

Hi, How we monitor our Streaming Analytics projects is by sending an email to a System Administrator if the CAS table we output to is missing or has not been updated with a new sensor record in the last 30 minutes. I created a SAS job and scheduled it to run in the SAS Job Execution Web application. You could check the status of a SQL table in your case. Good luck!