huggingface / huggingface/evaluate
Proposal: Add ECE to evaluate
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 341
- PR merge metrics
- No merged PRs in 30d
Description
Expected Calibration Error (ECE) is a widely-used metric for evaluating how well a classifier's predicted confidence scores align with actual outcomes (Guo et al., 2017). It is a standard diagnostic in modern deep learning pipelines, on par with Brier score (already in `evaluate`) for calibration assessment, but is not yet available in `evaluate`.
Happy to implement this myself, just wanted to check on scope and design before I open a PR.
## Proposed API
At minimum, this metric requires predictions and references as inputs.
```python
>>> import numpy as np
>>> ece = evaluate.load("ece")
>>> references = np.array([0, 0, 1, 1])
>>> predictions = np.array([0.25, 0.25, 0.75, 0.75])
>>> results = ece.compute(references=references, predictions=predictions,
n_bins=2)
>>> print(results)
{'ece': 0.25, 'mce': 0.25, 'adaptive_ece': 0.25}
```
### Inputs
- references: array-like of shape (n_samples,), representing the ground truth labels. Can be numeric (0/1 or -1/1) or strings.
- predictions: numeric array-like of shape (n_samples,), representing the predicted confidence scores (probabilities in [0, 1]) for the positive class.
### Optional arguments:
- n_bins: number of bins (default is 10).
- strategy: binning strategy "uniform" (equal-width) or "adaptive" (equal-mass) (default is "uniform").
- pos_label: int or str, default=None. Label of the positive class. pos_label will be inferred as follows:
- if references in {-1, 1} or {0, 1}, pos_label defaults to 1;
- else if references contains strings, pos_label must be explicitly specified (an error is raised otherwise);
- otherwise, pos_label defaults to the greater label, i.e. np.unique(references)[-1].
- return_detailed: if True, returns per-bin reliability diagram data (default is False).
### Output Values
This metric returns a dictionary with the following keys:
- ece (float): Expected Calibration Error.
- mce (float): Maximum Calibration Error.
- adaptive_ece (float): Adaptive ECE using equal-mass binning.
- reliability_diagram (list[dict], optional): per-bin data when return_detailed=True.
## Implementation approach
Brier score uses scikit-learn. ECE is not in `scikit-learn`, so either:
A. ECE is implemented from scratch using `numpy`
B. Or a dependency is added and the `netcal` implementation is used.
References
- [Guo et al., On Calibration of Modern Neural Networks, ICML 2017](https://arxiv.org/abs/1706.04599)
- [Nixon et al., Measuring Calibration in Deep Learning, CoRR 2020](https://arxiv.org/abs/1904.01685)
- netcal package: https://github.com/EFS-OpenSource/calibration-framework
Contributor guide
Research direction
Start with the existing Brier score metric and its scikit-learn integration, then compare the proposed inputs and outputs with Evaluate's metric conventions. Resolve whether to implement ECE with numpy or add netcal, define the binning and positive-label behavior, and validate the documented compute example plus detailed reliability-diagram output.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, python, scikit-learn
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100