huggingface / huggingface/lighteval

F1_score collapses nltk's undefined result onto zero, including when the gold reference is empty

Open
#1,353 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.5k
Forks
555
Avg merge
1d 6h
Merged PRs (30d)
1

Description

Small one, raised before I write about it in case I have misread it. Line numbers at commit `932e1f2f4c5af3e9`.

`src/lighteval/metrics/metrics_sample.py:238`, class `F1_score`:

```python
ret = nltk.scores.f_measure(gold_bow, pred_bow)
if ret is None:
return 0.0
```

nltk returns `None` from `f_measure` when either `precision` or `recall` is undefined. Those are two different situations:

- `precision` is undefined when the **prediction** set is empty — the documented case, and scoring it `0.0` is reasonable;
- `recall` is undefined when the **gold reference** set is empty — a dataset property, not a model failure.

nltk itself separates undefined from zero on adjacent lines: it returns `None` when a component is undefined, and returns `0` when `p` or `r` is defined-and-zero. Collapsing both to `0.0` means an empty gold reference is scored identically to a genuine miss, and nothing downstream can tell them apart.

## Suggestion

As an option: check the gold set explicitly before calling `f_measure` and surface empty references as excluded or as a dataset warning, rather than as a score.

## Disclosure

This appears as one instance in a short methods paper about evaluation instruments that state an invariant and do not apply it — most instances in it are from my own code. I would rather you saw it first, and if it is intended I will record that.

Contributor guide

No contributing guide indexed for this repository

Research direction

Read src/lighteval/metrics/metrics_sample.py at line 238 in F1_score, then verify how nltk.scores.f_measure distinguishes undefined precision and recall. Check how this metric's result is consumed; done means an empty gold reference is no longer indistinguishable from a genuine zero, with the chosen exclusion or warning behavior covered by the relevant existing checks.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.