Predicting abnormal trends in timeseries data

[duki]

I'm looking for an algorithm that can help identify abnormal trends in time-series metrics. The best I've been able to find so far is ARIMA (a completely new concept for me).

We offer several services which we monitor active usage against, for any given time of day (typically updated once a minute). Here's a example of one of our services:


What I'm interested in is gradual degradation that doesn't get noticed until ... well, it gets noticed. For example, if we zoom in to 12/6 we see:


So what I'm specifically interested in here is the trend downward `@17:30~`. Likewise another good example would be:


Here we can see the abnormalities on 12/1 and 12/4 during peak.

Are there any formal methods of detecting this sort of trend (or rather, abnormality to a historical trend)? Specifically interested in the _trend_ here, not the raw numbers. These services have a historical decreasing usage as time goes on - eventually, these numbers will drop from say 60k peaks to 15k peaks.

Is ARIMA the best bet here? e.g., predicting the future curve based on yesterday's curve, setting some threshold, and alerting if that threshold is breached? Or is there a more straightforward way to go about this? Am I fundamentally misunderstanding ARIMA?

 

Here's kind of what I had in mind for what I'd like to accomplish:

(i.e., taking the curve of one dataset and seeing if another dataset fits within that curve)

Edited December 10, 2015 by duki

[Klaynos]

My first guess would be using a machine learning method on the gradient of signal vs number of minutes (or seconds depends on sampling) past midnight.

[Prometheus]

I did a whole module on ARIMA but still don't understand it. Have you tried de-trending and eliminating seasonality from your data? What package are you using to analyse the data?

 

There are many other methods for analysing time series, but apparently ARIMA is the simplest so probably the best to start with.

[DevilSolution]

Looks like an ATR indicator to me. If its curved then use Fourier, but i have no idea what the data relates to in your image, looks more like quantity than price and volume has its own indicators with similar results.

 

My apologies my minds on other things, what exactly do you actively monitor? I wont hazard a guess.

 

An easy solution would be simple regression where the two line would meet at right angles, codes simple

Edited December 19, 2015 by DevilSolution

[Sensei]

12/2, 12/3, 12/4, etc. looks like date December 2,3,4,...

So one whole sinusoid half is one whole day.

And Y axis is quantity of people that used service. Right?

 

I would simply took newspaper, or tv program, and check in it what exactly happened 6 December and started around 18.

 

Could be football or other sport.

FIFA eliminations, EURO eliminations, local country league matches, etc.

French league had 3 matches 6 December.

 

Premier League, Everton vs Crystal Palace, match started 21:00, 7 December.

Primera Division, Barcelona vs Levante, started 20:30.

(We could be in different time zone, so you need to adjust to local time zone)

 

Are there any formal methods of detecting this sort of trend

 

Get newspaper, tv program, and check when there will be matches the next time...

 

Do you have access to per-country data?

If yes, check which country customers disappeared in this event at 6 Dec.

 

Do you have access to per-sex data? Most likely male users disappeared, if they were watching matches.

Edited December 24, 2015 by Sensei