huggingface / huggingface/lighteval
avg_at_n silently ignores strip_strings/normalize (preprocess not applied), deflating scores
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 555
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 1
Description
### Summary
`AvgAtN.compute` never applies `self.preprocess()`, so the `strip_strings` / `normalize` options the metric is configured with are silently ignored, and `avg@n` under-counts matches that differ only by whitespace or normalization.
`src/lighteval/metrics/metrics_sample.py` (AvgAtN.compute):
```python
all_scores = []
for i in range(self.n):
all_scores.append(self.compute_score(doc, model_response[i]))
```
The sibling sampling metrics both preprocess gold and predictions first: `MajAtN.compute` and `PassAtK.compute` build a new `Doc` from `self.preprocess(...)`-ed choices and call `self.preprocess(...)` on each prediction. `AvgAtN` does not, so it scores raw values.
The registered metric is built as `AvgAtN(strip_strings=True)` (`metrics.py`, `Metrics.avg_at_n`), and the default scorer does `gold == pred`. A generative prediction almost always carries trailing whitespace, so `"Paris\n"` scores 0 against gold `"Paris"`, and `avg@n` is systematically biased downward.
### Reproduction
```python
from lighteval.tasks.requests import Doc
from lighteval.models.model_output import ModelResponse
from lighteval.metrics.metrics_sample import AvgAtN
doc = Doc(query="q", choices=["London", "Paris", "Berlin"], gold_index=1)
resp = ModelResponse(text=["Paris\n", " Paris"])
print(AvgAtN(n=2, strip_strings=True).compute(doc, resp)) # 0.0, expected 1.0
```
`maj@n` and `pass@k` return 1.0 for the same input because they strip.
### Scope
Affects `avg_at_n` (default exact-match). `avg_at_n_math` is unaffected: it sets no `strip_strings`/`normalize`, so preprocessing is a no-op there.
### Fix
Apply `preprocess` to the gold and each prediction in `AvgAtN.compute`, mirroring `PassAtK` (preprocess `doc.choices`, keep `gold_index` so it stays valid). Happy to open a PR (with a regression test).
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in src/lighteval/metrics/metrics_sample.py at AvgAtN.compute and compare its preprocessing with MajAtN.compute and PassAtK.compute. Check the registered configuration in metrics.py and reproduce the trailing-whitespace case from the issue. Done means avg@n applies the configured preprocessing to gold choices and predictions while preserving gold_index, with a regression test showing the reported inputs score correctly.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 88/100