huggingface / huggingface/lighteval

[BUG] PMI normalization unusable: requires `unconditioned_logprob` but API/docs don’t expose a way to provide it

Open
#997 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
2.5k
Forks
555
Avg merge
1d 6h
Merged PRs (30d)
1

Description

## Describe the bug

`LogLikelihoodAccMetric(normalization=LogProbPMINorm())` appears as a supported normalization option, but it is currently **not usable**:

* When used directly, the pipeline asserts that `unconditioned_logprob` must be provided:

```
AssertionError: unconditioned_logprob must be provided for PMI normalization
```
* If I attempt to work around this by including both conditioned and unconditioned logprobs in the same list, the evaluation crashes with:

```
ValueError: choices_text and choices_logprob must have the same length.
```

There seems to be **no documented way to pass unconditioned logprobs** per choice, making PMI normalization effectively broken.

## To Reproduce

```bash
lighteval accelerate \
"model_name=Qwen/Qwen3-0.6B-Base"\
"community|readability_ceb_hybrid|0" \
--custom-tasks lighteval/community_tasks/filipino_evals.py \
--max-samples 20
```

OR

A minimal example illustrating the issue:

```python
from lighteval.metrics import LogLikelihoodAccMetric, LogProbPMINorm
from lighteval.pipeline import EvaluationPipeline

metric = LogLikelihoodAccMetric(normalization=LogProbPMINorm())

batch = {
"choices_text": ["A", "B"],
# Only conditioned logprobs are possible via current API:
"choices_logprob": [-1.2, -1.5],
# There's no supported field to provide unconditioned logprobs here.
}

pipe = EvaluationPipeline(metrics=[metric])
pipe.add_batch(batch)
pipe.evaluate()
# => AssertionError: unconditioned_logprob must be provided for PMI normalization
```

Attempting a workaround by adding both conditioned and unconditioned values to `choices_logprob` causes:

```
ValueError: choices_text and choices_logprob must have the same length.
```

## Expected behavior

One of the following should be supported:

1. **Documented input schema:**
Provide a clear way to supply unconditioned logprobs per choice.
For example:

```python
"choices_logprob": [-1.2, -1.5], # conditioned
"choices_uncond_logprob": [-0.9, -1.1] # unconditioned
```

2. **Automatic computation:**
If the backend can compute unconditioned logprobs automatically, PMI normalization should trigger that process internally.

3. **Better messaging / removal:**
If PMI normalization is not yet implemented, remove `LogProbPMINorm()` from the normalization options and update docs and error messages accordingly.

## Version info

* **OS:** [Ubuntu 22.04.3 LTS]
* **LightEval:** [0.11.0]
* **Backend:** [vllm,accelerate]
* **Python:** [3.10,3.12]

## Additional context / logs

* Error 1:

```
AssertionError: unconditioned_logprob must be provided for PMI normalization
```
* Error 2:

```
ValueError: choices_text and choices_logprob must have the same length.
```

[pmi.log](https://github.com/user-attachments/files/22598570/pmi.log)

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.