dotnet / dotnet/machinelearning

Log loss metric can be Infinity or NaN

Open
#2,708 2 comments 0 reactions 0 assignees View on GitHub
area-Core bug Priority:2
Dominant language
C#
Stars
9.4k
Forks
2k
Avg merge
2d 20h
Merged PRs (30d)
11

Description

For binary classification (and perhaps multiclass classification) the `log loss` can be infinite. The `log loss reduction` can also be negative infinity, as it is a shifting and rescaling of the `log loss`.

Similarly, the `log loss` can be a `NaN`. This is specifically guarded against in the code, but does seems like a bug too.

The culprit for both cases lies in the initial calculations in the `ProcessRow()` method of the `Aggregator` for the `BinaryClassifierEvaluator`.

```cs
Double logloss;
if (!Single.IsNaN(prob))
{
if (_label > 0)
{
// REVIEW: Should we bring back the option to use ln instead of log2?
logloss = -Math.Log(prob, 2);
}
else
logloss = -Math.Log(1.0 - prob, 2);
}
else
logloss = Double.NaN;
```

I propose that to guard against infinities we add an epsilon before taking the log.

To guard against `NaNs`, we will need to fix the probability calculations (e.g. in the calibrator(s)).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.