elastic / elastic/ml-cpp

[ML] Maximise minimum recall issues when classes are mainly separable

Open
#1,720 0 comments 0 reactions 0 assignees View on GitHub
:ml/DataFrameAnalysis >enhancement
Dominant language
C++
Stars
157
Forks
67
Avg merge
12h 48m
Merged PRs (30d)
16

Description

When the classes a largely separable all the predicted probabilities are close to either 0 or 1. In order to have any impact on class recall we then need to choose decision thresholds which are rather small or large. Moving the decision boundary so far can cause issues particularly when there are small domain shifts when the model is deployed. In such cases we should consider doing a couple of things:
1. Checking for errors in recall estimates (we can use the bagging we perform when training to get bootstrap estimates of these quantities). We should not adjust thresholds where we can't accurately measure recall.
2. Hard limit the maximum amount by which we'll adjust probabilities in order to impact improve minimum class recall.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.