dotnet / dotnet/machinelearning
Investigate thresholding binary log-loss
- Dominant language
- C#
- Stars
- 9.4k
- Forks
- 2k
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 11
Description
Multi-class log-loss has an `epsilon` value which thresholds the input probability between `epsilon` and `1.0`. This causes multi-class log-loss to never be `Infinity`.
The binary log-loss does not have an `epsilon` threshold, which causes the returned log-loss to be `Infinity` when the prediction is perfectly confident (p=0.0 or p=1.0) and incorrect about the true label.
Log-loss for binary classification:
https://github.com/dotnet/machinelearning/blob/062be280a9d6de23838d4db5ad93fea9d7d0c1f6/src/Microsoft.ML.Data/Evaluators/BinaryClassifierEvaluator.cs#L663-L677
Log-loss for multi-class:
https://github.com/dotnet/machinelearning/blob/062be280a9d6de23838d4db5ad93fea9d7d0c1f6/src/Microsoft.ML.Data/Evaluators/MulticlassClassificationEvaluator.cs#L450-L458
As part of thinking through the right behavior, we should investigate the behavior of other ML packages like scikit-learn/TensorFlow/PyTorch.
Contributor guide
Assessment
This issue has not been assessed yet.