huggingface / huggingface/evaluate
r_squared: documented zero_division parameter is not implemented; constant references return -inf/nan
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 341
- PR merge metrics
- No merged PRs in 30d
Description
### Describe the bug
`r_squared` documents a `zero_division` parameter that is not implemented, and the case it exists to handle silently returns `-inf` or `nan`.
The docstring in `_KWARGS_DESCRIPTION` says:
```
zero_division: Which value to substitute as a metric value when encountering zero division.
Should be one of 0, 1, "warn". "warn" acts as 0, but the warning is raised.
```
but the implementation is:
```python
def _compute(self, predictions=None, references=None):
```
`zero_division` appears exactly once in the file — in that docstring. Passing it raises `TypeError`.
### The underlying case
When every reference is identical there is no variance to explain, so `sst` is zero and R^2 is undefined. Nothing guards the division:
```python
sst = np.sum((references - mean_references) ** 2)
r_squared = 1 - (ssr / sst)
```
numpy returns `-inf` (or `nan` when predictions match exactly) with only a `RuntimeWarning`, and `round()` passes it straight through. Compared against `sklearn.metrics.r2_score`:
| predictions | references | `evaluate` | `sklearn.r2_score` |
|---|---|---|---|
| `[1.1, 2.1, 2.9, 4.2]` | `[1, 2, 3, 4]` | 0.986 | 0.986 |
| `[1, 2, 3, 4]` | `[5, 5, 5, 5]` | **-inf** | 0.0 |
| `[5, 5, 5, 5]` | `[5, 5, 5, 5]` | **nan** | 1.0 |
### Reproduction
```python
import evaluate
m = evaluate.load("r_squared")
m.compute(predictions=[1, 2, 3, 4], references=[5, 5, 5, 5]) # -inf
m.compute(predictions=[5, 5, 5, 5], references=[5, 5, 5, 5]) # nan
m.compute(predictions=[1, 2, 3, 4], references=[5, 5, 5, 5], zero_division=0)
# TypeError: _compute() got an unexpected keyword argument 'zero_division'
```
### Why it matters
Both values propagate. A single constant-reference batch turns an averaged score into `nan` for a whole run, and `-inf` dominates any mean. Constant references are not exotic — a held-out slice where the target happens not to vary, or a small evaluation group, is enough. Nothing raises, so the result looks like a modelling outcome rather than an undefined computation.
Unrelated but adjacent: `_DESCRIPTION` states "The R^2 value ranges from 0 to 1", which is not true for a fit worse than the mean — `sklearn` returns negative values there, as does this implementation.
### Environment
`evaluate` at `a7dd338`, numpy 1.26.4, scikit-learn 1.7.0. (Note the test suite does not run on Python 3.12+ — `tests/utils.py` imports `distutils`, removed in 3.12. Verified on 3.11.)
### Fix
PR to follow: implement `zero_division` as already documented, defaulting to `"warn"` so the currently silent case becomes visible rather than quietly changing to a value the caller did not choose.
Contributor guide
Research direction
Locate the r_squared metric implementation containing _KWARGS_DESCRIPTION and _compute, then inspect its existing tests and run the relevant test command. Implement the documented zero_division behavior with the default of "warn", and add coverage for constant references, including matching and non-matching predictions. Done means the parameter is accepted, undefined results follow the documented choice, and the inaccurate range description is addressed if in scope.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, python, scikit-learn
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100