tensorflow / tensorflow/probability

Maybe incorrect Brier score calculation

Open
#1,327 1 comment 0 reactions 1 assignee View on GitHub

@srvasude is already working on this.

Since May 5, 2021.

Dominant language
Jupyter Notebook
Stars
4.4k
Forks
1.1k
PR merge metrics
No merged PRs in 30d

Description

From https://github.com/tensorflow/tensorflow/issues/41664

System information

  • Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes
  • TensorFlow version (use command below): 1.15.0
  • TensorFlow Probability version: 0.8.0

Describe the current behavior

I believe that the way the Brier score calculation is implemented in tensorflow_probability.stats.brier_score is incorrect. The formula used is

sum_i p[i]* p[i] - 2*p[k]

where p is the probability vector over all discrete outcomes, and k is the realized outcome. (Note: This gives element wise Brier scores, and to get the actual Brier score across the dataset, reduce_mean() must be called.)

Describe the expected behavior

This formula does not match the reference cited nor Wikipedia nor the definition that sklearn uses. Also, it is stated in the tensorflow_probability docs that the Brier score can be negative, which is not true.

The reference cited is Brier’s original paper, found here, which states the formula as

Screen Shot 2020-07-23 at 09 48 34

where r is the number of possible classes, n is the number of forecasts, fij is the probability forecast of class j for instance i, and Eij is 0 or 1 depending if the event occurred in class j or not. This is the same formula that is used in sklearn and Wikipedia. By definition, this score cannot be negative.

If we consider the element wise Brier score from this formu

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.