dotnet / dotnet/machinelearning

Investigate thresholding binary log-loss

Open
#5,055 2 comments 1 reaction 0 assignees View on GitHub
area-Core Priority:2
Dominant language
C#
Stars
9.4k
Forks
2k
Avg merge
2d 20h
Merged PRs (30d)
11

Description

Multi-class log-loss has an `epsilon` value which thresholds the input probability between `epsilon` and `1.0`. This causes multi-class log-loss to never be `Infinity`.

The binary log-loss does not have an `epsilon` threshold, which causes the returned log-loss to be `Infinity` when the prediction is perfectly confident (p=0.0 or p=1.0) and incorrect about the true label.

Log-loss for binary classification:
https://github.com/dotnet/machinelearning/blob/062be280a9d6de23838d4db5ad93fea9d7d0c1f6/src/Microsoft.ML.Data/Evaluators/BinaryClassifierEvaluator.cs#L663-L677

Log-loss for multi-class:
https://github.com/dotnet/machinelearning/blob/062be280a9d6de23838d4db5ad93fea9d7d0c1f6/src/Microsoft.ML.Data/Evaluators/MulticlassClassificationEvaluator.cs#L450-L458

As part of thinking through the right behavior, we should investigate the behavior of other ML packages like scikit-learn/TensorFlow/PyTorch.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.