huggingface / huggingface/lighteval

SamplingMetric string `normalize` argument always raises: getmembers() list treated as a dict

Open Beginner friendly
#1,365 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.5k
Forks
555
Avg merge
1d 6h
Merged PRs (30d)
1

Description

## Describe the bug

`SamplingMetric.__init__` (`src/lighteval/metrics/metrics_sample.py`, around lines 1121-1128) resolves a string `normalize` argument like this:

```python
allowed_normalizations = inspect.getmembers(
lighteval.metrics.normalizations, inspect.isfunction
) # -> {name: fn}
if normalize in allowed_normalizations:
self.normalize = allowed_normalizations[normalize]
else:
raise ValueError(f"Unknown normalization function: {normalize}")
```

`inspect.getmembers(...)` returns a list of `(name, function)` tuples, not a dict (the trailing comment is inaccurate). As a result:

- `normalize in allowed_normalizations` compares a string against tuples and is always `False`, even for a valid function name.
- The lookup `allowed_normalizations[normalize]` would raise `TypeError: list indices must be integers or slices, not str`, but it is never reached because the `if` is always `False`.

So the documented `normalize: Callable | str | None` argument is broken for the `str` form on every sampling metric (`AvgAtN`, `MajAtN`, `PassAtK`, `GPassAtK`): passing a valid normalizer name raises `ValueError: Unknown normalization function: ...`.

## To reproduce

Pure standard library, no model call:

```python
import inspect
import lighteval.metrics.normalizations as norm

allowed = inspect.getmembers(norm, inspect.isfunction)
print(type(allowed).__name__) # list
print("helm_normalizer" in allowed) # False, although helm_normalizer is a valid normalizer
allowed["helm_normalizer"] # TypeError: list indices must be integers or slices, not str
```

Or through the public API:

```python
from lighteval.metrics.metrics_sample import PassAtK

PassAtK(normalize="helm_normalizer") # ValueError: Unknown normalization function: helm_normalizer
```

## Expected behavior

A valid normalizer name resolves to the corresponding function; only an unknown name raises `ValueError`.

## Suggested fix

Build a dict from the members:

```python
allowed_normalizations = dict(
inspect.getmembers(lighteval.metrics.normalizations, inspect.isfunction)
)
```

`normalize in allowed_normalizations` then checks names, and `allowed_normalizations[normalize]` returns the function. Happy to open a PR with this plus a unit test.

## Version

main (commit 932e1f2f4c5af3e926534f12b2a84a3ae18d6d3f).

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in src/lighteval/metrics/metrics_sample.py around lines 1121-1128 and inspect how SamplingMetric resolves string normalizer names from lighteval.metrics.normalizations. Add coverage for a valid string passed through PassAtK and for an unknown name, then run the relevant unit tests to confirm valid names resolve and unknown names still raise ValueError.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing-qa
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
88/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.