huggingface / huggingface/lighteval

avg_at_n silently ignores strip_strings/normalize (preprocess not applied), deflating scores

Open Beginner friendly
#1,384 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.5k
Forks
555
Avg merge
1d 6h
Merged PRs (30d)
1

Description

### Summary

`AvgAtN.compute` never applies `self.preprocess()`, so the `strip_strings` / `normalize` options the metric is configured with are silently ignored, and `avg@n` under-counts matches that differ only by whitespace or normalization.

`src/lighteval/metrics/metrics_sample.py` (AvgAtN.compute):

```python
all_scores = []
for i in range(self.n):
all_scores.append(self.compute_score(doc, model_response[i]))
```

The sibling sampling metrics both preprocess gold and predictions first: `MajAtN.compute` and `PassAtK.compute` build a new `Doc` from `self.preprocess(...)`-ed choices and call `self.preprocess(...)` on each prediction. `AvgAtN` does not, so it scores raw values.

The registered metric is built as `AvgAtN(strip_strings=True)` (`metrics.py`, `Metrics.avg_at_n`), and the default scorer does `gold == pred`. A generative prediction almost always carries trailing whitespace, so `"Paris\n"` scores 0 against gold `"Paris"`, and `avg@n` is systematically biased downward.

### Reproduction

```python
from lighteval.tasks.requests import Doc
from lighteval.models.model_output import ModelResponse
from lighteval.metrics.metrics_sample import AvgAtN

doc = Doc(query="q", choices=["London", "Paris", "Berlin"], gold_index=1)
resp = ModelResponse(text=["Paris\n", " Paris"])
print(AvgAtN(n=2, strip_strings=True).compute(doc, resp)) # 0.0, expected 1.0
```

`maj@n` and `pass@k` return 1.0 for the same input because they strip.

### Scope

Affects `avg_at_n` (default exact-match). `avg_at_n_math` is unaffected: it sets no `strip_strings`/`normalize`, so preprocessing is a no-op there.

### Fix

Apply `preprocess` to the gold and each prediction in `AvgAtN.compute`, mirroring `PassAtK` (preprocess `doc.choices`, keep `gold_index` so it stays valid). Happy to open a PR (with a regression test).

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in src/lighteval/metrics/metrics_sample.py at AvgAtN.compute and compare its preprocessing with MajAtN.compute and PassAtK.compute. Check the registered configuration in metrics.py and reproduce the trailing-whitespace case from the issue. Done means avg@n applies the configured preprocessing to gold choices and predictions while preserving gold_index, with a regression test showing the reported inputs score correctly.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
88/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.