huggingface / huggingface/lighteval
[BUG] bug prevents mixing batched and non-batched metrics
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 555
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 1
Description
## Describe the bug
The `apply_metric` function in `lighteval/metrics/__init__.py` has a bug that prevents mixing batched and non-batched metrics. When both types of metrics are present in the same evaluation, the function fails to correctly compute and combine results from both metric types, leading to incomplete or incorrect metric outputs.
## Root Cause
The `apply_metric` function has a logical flaw: it uses a single `for metric in metrics:` loop with an `if/else` statement that only executes one branch per function call. If the first metric is batched, it will process all batched metrics but completely skip all non-batched metrics. If the first metric is non-batched, it will try to process all metrics (including batched ones) as non-batched, calling them incorrectly and ignoring their batching requirements. The function currently assumes all metrics would be of the same type rather than handling mixed batched/non
## To Reproduce
1. Create a list of metrics that includes both batched metrics (where `metric.batched_compute = True`) and non-batched metrics (where `metric.batched_compute = False`)
2. Call `apply_metric(responses, docs, metrics)` with this mixed list of metrics
3. Observe that the function either:
- Only processes one type of metric (batched or non-batched)
- Produces incorrect output structure
- Fails to combine results from both metric types properly
**Minimal working example:**
```python
from lighteval.metrics import apply_metric
# Assume we have:
# - batched_metric with batched_compute = True
# - non_batched_metric with batched_compute = False
metrics = [batched_metric, non_batched_metric]
# This will not work correctly with the buggy version
results = apply_metric(responses, docs, metrics)
```
## Expected behavior
The function should correctly handle a mixed list of batched and non-batched metrics by:
1. Separating metrics into batched and non-batched groups
2. Computing batched metrics once for all samples
3. Computing non-batched metrics individually for each sample
4. Combining results from both types into a unified output structure where each sample gets results from all metrics
The output should be a list of dictionaries, one per sample, containing the combined results from all metrics regardless of their batching behavior.
## Version info
- Operating system: Linux
- lighteval version: commit 7ed2636ed53142c0cf5e4aa752970ae62bcf1c24 (before the fix)
- Environment: pip/conda environment with lighteval
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.