huggingface / huggingface/lighteval
F1_score collapses nltk's undefined result onto zero, including when the gold reference is empty
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 555
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 1
Description
Small one, raised before I write about it in case I have misread it. Line numbers at commit `932e1f2f4c5af3e9`.
`src/lighteval/metrics/metrics_sample.py:238`, class `F1_score`:
```python
ret = nltk.scores.f_measure(gold_bow, pred_bow)
if ret is None:
return 0.0
```
nltk returns `None` from `f_measure` when either `precision` or `recall` is undefined. Those are two different situations:
- `precision` is undefined when the **prediction** set is empty — the documented case, and scoring it `0.0` is reasonable;
- `recall` is undefined when the **gold reference** set is empty — a dataset property, not a model failure.
nltk itself separates undefined from zero on adjacent lines: it returns `None` when a component is undefined, and returns `0` when `p` or `r` is defined-and-zero. Collapsing both to `0.0` means an empty gold reference is scored identically to a genuine miss, and nothing downstream can tell them apart.
## Suggestion
As an option: check the gold set explicitly before calling `f_measure` and surface empty references as excluded or as a dataset warning, rather than as a score.
## Disclosure
This appears as one instance in a short methods paper about evaluation instruments that state an invariant and do not apply it — most instances in it are from my own code. I would rather you saw it first, and if it is intended I will record that.
Contributor guide
No contributing guide indexed for this repository
Research direction
Read src/lighteval/metrics/metrics_sample.py at line 238 in F1_score, then verify how nltk.scores.f_measure distinguishes undefined precision and recall. Check how this metric's result is consumed; done means an empty gold reference is no longer indistinguishable from a genuine zero, with the chosen exclusion or warning behavior covered by the relevant existing checks.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 58/100