elastic / elastic/ml-cpp

[ML] Improve the usability of `skip_model_update` rules

Open
#1,272 0 comments 0 reactions 0 assignees View on GitHub
:ml >enhancement
Dominant language
C++
Stars
157
Forks
67
Avg merge
12h 48m
Merged PRs (30d)
16

Description

`skip_model_update` is potentially a powerful technique for excluding updates to the model by known bad events, but it has some usability issues:
1. If you apply it from the start of the modelling it can stop the model learning anything at all.
2. It can stop the model ever adapting to some change in data characteristics.
3. It doesn't stop score quantiles being updated. This means anomaly severities from excluded values can diminish over time. However, if you are told something should be excluded because it is a known bad event you shouldn't decrease its severity based on frequency.
4. You can only apply it based on absolute thresholds, such as the magnitude of the difference from prediction. A more natural approach would be to apply it based on the anomaly score or some related measure.

Note that 4 mitigates problem 1 to some extent; we could also have a fade in. Problem 2 could be addressed if we made the model mainly (but not completely) ignore the skipped data. This would probably give an all round nicer experience.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.