huggingface / huggingface/lighteval
Normalized multiple-choice probability underflows for long sequence log-probabilities
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 555
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 1
Description
## Bug
`NormalizedMultiChoiceProbability.compute()` exponentiates sequence
log-probabilities before normalizing them. When every choice has a sufficiently
negative but finite log-probability, each value underflows to zero and the
metric returns `0.0` instead of the gold choice's relative probability.
This is easy to reach for long multiple-choice continuations because the metric
receives sequence-level log-likelihoods, not token probabilities.
## Reproduction
No model or API call is required:
```python
from lighteval.metrics.dynamic_metrics import NormalizedMultiChoiceProbMetric
from lighteval.models.model_output import ModelResponse
from lighteval.tasks.requests import Doc
doc = Doc(query="q", choices=["A", "B"], gold_index=0, task_name="test")
result = NormalizedMultiChoiceProbMetric().compute_sample(
doc=doc,
model_response=ModelResponse(logprobs=[-1000.0, -1001.0]),
)
print(result["normalized_mc_prob"])
```
On current `main` at
`932e1f2f4c5af3e926534f12b2a84a3ae18d6d3f`, this prints:
```text
0.0
```
and emits `RuntimeWarning: invalid value encountered in divide`.
The stable expected value is:
```python
1 / (1 + exp(-1)) == 0.7310585786300049
```
## Root cause
The current path is:
```python
normalized_probs = np.exp(normalized_log_probs)
normalized_probs = safe_divide(
normalized_probs[gold_ixs],
np.sum(normalized_probs),
)
```
For `[-1000, -1001]`, both exponentials are `0.0`, so the denominator is also
zero. The fallback in `safe_divide` then silently converts the metric to zero.
## Proposed fix
For finite values, subtract the largest normalized log-probability before
exponentiating. Subtracting one shared constant preserves every probability
ratio while preventing underflow:
```python
log_probs = np.asarray(normalized_log_probs, dtype=np.float64)
if log_probs.size > 0:
max_log_prob = np.max(log_probs)
if np.isfinite(max_log_prob):
log_probs = log_probs - max_log_prob
```
A focused regression can use `[-1000.0, -1001.0]` and compare with the
independent logistic expression above. Existing finite normal-range behavior,
multi-gold aggregation, and character/token/PMI normalization remain unchanged.
The all-`-inf`, `NaN`, `+inf`, and empty-choice cases are intentionally outside
this report; the finite shift leaves their pre-existing behavior unchanged.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at NormalizedMultiChoiceProbability.compute() in lighteval.metrics.dynamic_metrics and run the provided [-1000.0, -1001.0] reproduction. Add a focused regression for finite long-sequence log-probabilities, preserving existing normalization behavior; done means the result is approximately 0.7310585786300049 without the underflow warning.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, python
- Domain
- machine-learning, testing
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 78/100