[ML] Improve the usability of `skip_model_update` rules
- Vorherrschende Sprache
- C++
- Sterne
- 157
- Forks
- 67
- Ø Merge
- 17 Std. 52 Min.
- Gemergte PRs (30 T.)
- 20
Beschreibung
`skip_model_update` is potentially a powerful technique for excluding updates to the model by known bad events, but it has some usability issues:
1. If you apply it from the start of the modelling it can stop the model learning anything at all.
2. It can stop the model ever adapting to some change in data characteristics.
3. It doesn't stop score quantiles being updated. This means anomaly severities from excluded values can diminish over time. However, if you are told something should be excluded because it is a known bad event you shouldn't decrease its severity based on frequency.
4. You can only apply it based on absolute thresholds, such as the magnitude of the difference from prediction. A more natural approach would be to apply it based on the anomaly score or some related measure.
Note that 4 mitigates problem 1 to some extent; we could also have a fade in. Problem 2 could be addressed if we made the model mainly (but not completely) ignore the skipped data. This would probably give an all round nicer experience.
Beitragsleitfaden
Rechercherichtung
Das Issue nennt keine Implementierungsdateien, Tests oder einen Einstiegspunkt. Beginne damit, die Behandlung der Regel `skip_model_update` sowie die Pfade für Modellaktualisierung und Score-Quantile zu lokalisieren, und kläre anschließend, welche Änderung an der Benutzerfreundlichkeit beabsichtigt ist. Die Abschlusskriterien erfordern ein abgestimmtes Design, das die vier aufgeführten Aspekte und entsprechende Tests abdeckt.
Vom Indexierungsmodell aus dem Issue-Text verfasst.
Bewertung
- Tech-Stack
- cpp
- Bereich
- machine-learning
- Issue-Typ
- Feature
- Schwierigkeit
- 5/5
- Geschätzter Aufwand
- Über eine Woche
- Aktivitätsstatus
- Veraltet
- Klarheit
- Muss geklärt werden
- Anfängerfreundlichkeit
- 25/100