can we use in sas ESp SST to detect anomalies if it is timeseries data?

[Hansdewit]

we are trying to solve anomaly detection problem and are going to use sas ESP!  For this case it is time series data. 

We have every 5 minutes new data and the data has a daily pattern.  Can we use in Sas ESP using subspace tracking (SST) to identify the anomalies?

[sbxkoenk]

Hello,

Not sure what you want to know.

Are you asking if SubSpace Tracking (SST) is available in SAS ESP as an online model?

The answer is YES (as from a certain version at least).

See for example:

SAS® Event Stream Processing 6.1: Using Streaming Analytics
https://go.documentation.sas.com/doc/en/espcdc/6.1/espan/titlepage.htm

(CTRL-F 'SST')

This is ESP 6.1 but I know earlier and later releases of ESP have SST as well.

Cheers,

Koen

[sbxkoenk]

Additionally, You can search for 'SST' in Learn (Ask the Expert) and Community (SAS Community Library).

You will find some articles about ESP and SST (or better: SST in ESP).

Koen

[TomTuning]

It sure can and we have 2 great github entries to guide your way.  Check these out: 

https://github.com/sassoftware/iot-anomaly-detection-floodlights

https://github.com/sassoftware/iot-anomaly-detection-from-mobile-data

[Hansdewit]

this how my data looks like. 

this is how my data looks like

[sbxkoenk]

Hello Hans,

I see.

I know how to detect such time series anomaly when using 'regular' SAS (offline), but I don't know how to set up ESP to catch this weird behavior of the time series at 12MAR2021 just before noon.

If you want more info on anomaly detection in time series using SAS/ETS and SAS Visual Forecasting procedures, then let me know.

If your only interest is online TS anomaly detection (with ESP) then ESP experts should and will chime in.

Good luck,

Koen

[Hansdewit]

my data is every 5 minutes, so we can do it offline. 

at this moment we don't have a license of Sas Visual Forecasting, but in our Sas 9.4 environment we have ETS.  If you can give a direction, that will be fantastic. Timeseries is new for me.

[sbxkoenk · Accepted solution]

Hello,

While Visual Forecasting offers nice extra features for anomaly detection in Time Series (like the Time Series Motif Discovery Package), lots can be done with SAS/ETS as well (also in SAS 9.4).

You can search for any anomaly and / or structural change with SAS/ETS:

• additive outliers
• changing level (permanent or temporary, sudden or slow)
• changing trend (sudden rupture or slow change)
• changing slope
• changing seasonality
• changing variance (dispersion)
• changing influence of explanatory inputs (time-varying regression coefficients)
• changing ...

In SAS/ETS (contrary to ESP!), you mostly need to see the full series for some anomalies to be detected. So you can easily find anomalies and structural changes in historical patterns over time (looking backward). In SAS/ETS it's not so straightforward (euphemistically said) to discover anomalies in NEW points (last point that was added to the series = detection while you are moving forward).

Anomalies and structural changes can be detected with PROC ARIMA, PROC UCM, PROC SSM (State Space Models), PROC X11 & X12 & X13 and so on ... Also SAS/STAT can be used by 'tweaking' some procedures like PROC ADAPTIVEREG. 

In most cases you need to make a TS-model in order for outliers and breaks to be detected.

If you search the SAS/ETS doc for 'outlier' or 'break' you will find useful examples for sure.

Good luck,

Koen

[sbxkoenk]

Hello,

I also want to emphasize you are wanting to detect a temporary change rather than a permanent change in your time series pattern (course over time).

It's possible with SAS/ETS of course but it's really a pity you don't have:

SAS Viya Forecasting --> Time Series Mining. In that package you have a task called 'Subsequence Anomaly Detection' and that's really what you are after.

Cheers,

Koen

[Hansdewit]

as data scientist I want Sas Visual forecasting. But I am not paying the bills.