huggingface / huggingface/evaluate

r_squared: documented zero_division parameter is not implemented; constant references return -inf/nan

Open
#803 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.5k
Forks
341
PR merge metrics
No merged PRs in 30d

Description

### Describe the bug

`r_squared` documents a `zero_division` parameter that is not implemented, and the case it exists to handle silently returns `-inf` or `nan`.

The docstring in `_KWARGS_DESCRIPTION` says:

```
zero_division: Which value to substitute as a metric value when encountering zero division.
Should be one of 0, 1, "warn". "warn" acts as 0, but the warning is raised.
```

but the implementation is:

```python
def _compute(self, predictions=None, references=None):
```

`zero_division` appears exactly once in the file — in that docstring. Passing it raises `TypeError`.

### The underlying case

When every reference is identical there is no variance to explain, so `sst` is zero and R^2 is undefined. Nothing guards the division:

```python
sst = np.sum((references - mean_references) ** 2)
r_squared = 1 - (ssr / sst)
```

numpy returns `-inf` (or `nan` when predictions match exactly) with only a `RuntimeWarning`, and `round()` passes it straight through. Compared against `sklearn.metrics.r2_score`:

| predictions | references | `evaluate` | `sklearn.r2_score` |
|---|---|---|---|
| `[1.1, 2.1, 2.9, 4.2]` | `[1, 2, 3, 4]` | 0.986 | 0.986 |
| `[1, 2, 3, 4]` | `[5, 5, 5, 5]` | **-inf** | 0.0 |
| `[5, 5, 5, 5]` | `[5, 5, 5, 5]` | **nan** | 1.0 |

### Reproduction

```python
import evaluate
m = evaluate.load("r_squared")
m.compute(predictions=[1, 2, 3, 4], references=[5, 5, 5, 5]) # -inf
m.compute(predictions=[5, 5, 5, 5], references=[5, 5, 5, 5]) # nan
m.compute(predictions=[1, 2, 3, 4], references=[5, 5, 5, 5], zero_division=0)
# TypeError: _compute() got an unexpected keyword argument 'zero_division'
```

### Why it matters

Both values propagate. A single constant-reference batch turns an averaged score into `nan` for a whole run, and `-inf` dominates any mean. Constant references are not exotic — a held-out slice where the target happens not to vary, or a small evaluation group, is enough. Nothing raises, so the result looks like a modelling outcome rather than an undefined computation.

Unrelated but adjacent: `_DESCRIPTION` states "The R^2 value ranges from 0 to 1", which is not true for a fit worse than the mean — `sklearn` returns negative values there, as does this implementation.

### Environment

`evaluate` at `a7dd338`, numpy 1.26.4, scikit-learn 1.7.0. (Note the test suite does not run on Python 3.12+ — `tests/utils.py` imports `distutils`, removed in 3.12. Verified on 3.11.)

### Fix

PR to follow: implement `zero_division` as already documented, defaulting to `"warn"` so the currently silent case becomes visible rather than quietly changing to a value the caller did not choose.

Contributor guide

Open the contributing guide

Research direction

Locate the r_squared metric implementation containing _KWARGS_DESCRIPTION and _compute, then inspect its existing tests and run the relevant test command. Implement the documented zero_division behavior with the default of "warn", and add coverage for constant references, including matching and non-matching predictions. Done means the parameter is accepted, undefined results follow the documented choice, and the inaccurate range description is addressed if in scope.

Written by the indexing model from the issue text.

Assessment

Tech stack
numpy, python, scikit-learn
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.