tensorflow / tensorflow/probability
Maybe incorrect Brier score calculation
@srvasude is already working on this.
Since May 5, 2021.
- Dominant language
- Jupyter Notebook
- Stars
- 4.4k
- Forks
- 1.1k
- PR merge metrics
- No merged PRs in 30d
Description
From https://github.com/tensorflow/tensorflow/issues/41664
System information
- Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes
- TensorFlow version (use command below): 1.15.0
- TensorFlow Probability version: 0.8.0
Describe the current behavior
I believe that the way the Brier score calculation is implemented in tensorflow_probability.stats.brier_score is incorrect. The formula used is
sum_i p[i]* p[i] - 2*p[k]
where p is the probability vector over all discrete outcomes, and k is the realized outcome. (Note: This gives element wise Brier scores, and to get the actual Brier score across the dataset, reduce_mean() must be called.)
Describe the expected behavior
This formula does not match the reference cited nor Wikipedia nor the definition that sklearn uses. Also, it is stated in the tensorflow_probability docs that the Brier score can be negative, which is not true.
The reference cited is Brier’s original paper, found here, which states the formula as
![]()
where r is the number of possible classes, n is the number of forecasts, fij is the probability forecast of class j for instance i, and Eij is 0 or 1 depending if the event occurred in class j or not. This is the same formula that is used in sklearn and Wikipedia. By definition, this score cannot be negative.
If we consider the element wise Brier score from this formu
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.