topic Re: can we use in sas ESp SST to detect anomalies if it is timeseries data? in Streaming Analytics

can we use in sas ESp SST to detect anomalies if it is timeseries data?

Hansdewit — Mon, 03 May 2021 11:59:02 GMT

we are trying to solve anomaly detection problem and are going to use sas ESP! For this case it is time series data.

We have every 5 minutes new data and the data has a daily pattern. Can we use in Sas ESP using subspace tracking (SST) to identify the anomalies?

Re: can we use in sas ESp SST to detect anomalies if it is timeseries data?

sbxkoenk — Mon, 03 May 2021 15:20:11 GMT

Hello,

Not sure what you want to know.

Are you asking if SubSpace Tracking (SST) is available in SAS ESP as an online model?

The answer is YES (as from a certain version at least).

See for example:

SAS® Event Stream Processing 6.1: Using Streaming Analytics
https://go.documentation.sas.com/doc/en/espcdc/6.1/espan/titlepage.htm

(CTRL-F 'SST')

This is ESP 6.1 but I know earlier and later releases of ESP have SST as well.

Cheers,

Koen

Re: can we use in sas ESp SST to detect anomalies if it is timeseries data?

sbxkoenk — Mon, 03 May 2021 15:31:44 GMT

Additionally, You can search for 'SST' in Learn (Ask the Expert) and Community (SAS Community Library).

You will find some articles about ESP and SST (or better: SST in ESP).

Koen

Re: can we use in sas ESp SST to detect anomalies if it is timeseries data?

TomTuning — Mon, 03 May 2021 19:23:31 GMT

It sure can and we have 2 great github entries to guide your way. Check these out:

https://github.com/sassoftware/iot-anomaly-detection-floodlights

https://github.com/sassoftware/iot-anomaly-detection-from-mobile-data

Re: can we use in sas ESp SST to detect anomalies if it is timeseries data?

Hansdewit — Tue, 04 May 2021 06:58:01 GMT

this how my data looks like.

this is how my data looks like

Re: can we use in sas ESp SST to detect anomalies if it is timeseries data?

sbxkoenk — Tue, 04 May 2021 07:55:16 GMT

Hello Hans,

I see.

I know how to detect such time series anomaly when using 'regular' SAS (offline), but I don't know how to set up ESP to catch this weird behavior of the time series at 12MAR2021 just before noon.

If you want more info on anomaly detection in time series using SAS/ETS and SAS Visual Forecasting procedures, then let me know.

If your only interest is online TS anomaly detection (with ESP) then ESP experts should and will chime in.

Good luck,

Koen

Re: can we use in sas ESp SST to detect anomalies if it is timeseries data?

Hansdewit — Tue, 04 May 2021 09:01:18 GMT

my data is every 5 minutes, so we can do it offline.

at this moment we don't have a license of Sas Visual Forecasting, but in our Sas 9.4 environment we have ETS. If you can give a direction, that will be fantastic. Timeseries is new for me.

Re: can we use in sas ESp SST to detect anomalies if it is timeseries data?

sbxkoenk — Tue, 04 May 2021 12:25:16 GMT

Hello,

While Visual Forecasting offers nice extra features for anomaly detection in Time Series (like the Time Series Motif Discovery Package), lots can be done with SAS/ETS as well (also in SAS 9.4).

You can search for any anomaly and / or structural change with SAS/ETS:

additive outliers
changing level (permanent or temporary, sudden or slow)
changing trend (sudden rupture or slow change)
changing slope
changing seasonality
changing variance (dispersion)
changing influence of explanatory inputs (time-varying regression coefficients)
changing ...

In SAS/ETS (contrary to ESP!), you mostly need to see the full series for some anomalies to be detected. So you can easily find anomalies and structural changes in historical patterns over time (looking backward). In SAS/ETS it's not so straightforward (euphemistically said) to discover anomalies in NEW points (last point that was added to the series = detection while you are moving forward).

Anomalies and structural changes can be detected with PROC ARIMA, PROC UCM, PROC SSM (State Space Models), PROC X11 & X12 & X13 and so on ... Also SAS/STAT can be used by 'tweaking' some procedures like PROC ADAPTIVEREG.

In most cases you need to make a TS-model in order for outliers and breaks to be detected.

If you search the SAS/ETS doc for 'outlier' or 'break' you will find useful examples for sure.

Good luck,

Koen

Re: can we use in sas ESp SST to detect anomalies if it is timeseries data?

sbxkoenk — Wed, 05 May 2021 07:14:13 GMT

Hello,

I also want to emphasize you are wanting to detect a temporary change rather than a permanent change in your time series pattern (course over time).

It's possible with SAS/ETS of course but it's really a pity you don't have:

SAS Viya Forecasting --> Time Series Mining. In that package you have a task called 'Subsequence Anomaly Detection' and that's really what you are after.

Cheers,

Koen

Re: can we use in sas ESp SST to detect anomalies if it is timeseries data?

Hansdewit — Wed, 05 May 2021 11:26:11 GMT

as data scientist I want Sas Visual forecasting. But I am not paying the bills.