elastic / elastic/ml-cpp

[ML] Improve handling of bucket count variation for mean value anomaly detection

Offen
#1,386 4 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
:ml >enhancement
Vorherrschende Sprache
C++
Sterne
157
Forks
67
Ø Merge
17 Std. 52 Min.
Gemergte PRs (30 T.)
20

Beschreibung

Currently, we use a worst case estimate of the impact of changes in the count of values in a bucket on their mean variance. This is safe in the sense of not generating false positives, but can lead to large changes in the model plot bounds and potentially false negatives when the count of values in the bucket is low.

Specifically, we assume all measurements are independent so that the variance of the mean statistic will be proportional to 1 / "number samples" in the bucket. If the rate of values is highly variable this can lead to large increases in the width of the model bounds when the count is low. Unfortunately, this is not calibrated to the actual data behaviour. For example, in the other extreme, if all measurements in each bucket were perfectly correlated then we would get no change in variation of the mean statistic as a function of bucket count.

It would be possible to estimate the relationship between the bucket count and the sufficient statistics related to data variation for all the residual distributions we fit since we know the sample count for each bucket. This would also be a more accurate way of calibrating heavy tailed distributions like the log-normal to observed changes in the seasonal variation.

A computationally feasible formulation would be to use linear regression. For example, for the normal model we could fit the linear model `(x_i - m)^2 = [c_i s_i] [p_1 p_2]^t` for parameters `p_1` and `p_2`, observed bucket values `x_i`, mean of `x_i` `m` and bucket count and seasonal variance scale `c_i` and `s_i`, respectively. If we solve this in the least squares sense we only need to maintain a small set of statistics rather than all bucket values, which is the key to this being tractable for us in the streaming setting. The same formulation carries over for the log mean and log variance we estimate for the log-normal distribution and so on.

Beitragsleitfaden

Beitragsleitfaden öffnen

Rechercherichtung

Im Issue ist keine Datei, kein Test und kein Einstiegspunkt genannt. Beginne damit, den Code für die Residuenverteilung und die hinreichenden Statistiken des Mean-Value-Anomaliedetektors zu lokalisieren, und untersuche anschließend, wie Bucket-Anzahl und saisonale Varianzskalierung in die Normal- und Lognormalmodelle eingehen. Als abgeschlossen gilt die Aufgabe, wenn ein rechnerisch praktikabler Streaming-Ansatz vorhanden ist, der die Variation anhand der Bucket-Anzahlen kalibriert, ohne alle Bucket-Werte zu speichern.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
cpp, machine-learning
Bereich
machine-learning
Issue-Typ
Feature
Schwierigkeit
5/5
Geschätzter Aufwand
Über eine Woche
Aktivitätsstatus
Veraltet
Klarheit
Muss geklärt werden
Anfängerfreundlichkeit
25/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.