huggingface / huggingface/lighteval
[BUG] PMI normalization unusable: requires `unconditioned_logprob` but API/docs don’t expose a way to provide it
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 555
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 1
Description
## Describe the bug
`LogLikelihoodAccMetric(normalization=LogProbPMINorm())` appears as a supported normalization option, but it is currently **not usable**:
* When used directly, the pipeline asserts that `unconditioned_logprob` must be provided:
```
AssertionError: unconditioned_logprob must be provided for PMI normalization
```
* If I attempt to work around this by including both conditioned and unconditioned logprobs in the same list, the evaluation crashes with:
```
ValueError: choices_text and choices_logprob must have the same length.
```
There seems to be **no documented way to pass unconditioned logprobs** per choice, making PMI normalization effectively broken.
## To Reproduce
```bash
lighteval accelerate \
"model_name=Qwen/Qwen3-0.6B-Base"\
"community|readability_ceb_hybrid|0" \
--custom-tasks lighteval/community_tasks/filipino_evals.py \
--max-samples 20
```
OR
A minimal example illustrating the issue:
```python
from lighteval.metrics import LogLikelihoodAccMetric, LogProbPMINorm
from lighteval.pipeline import EvaluationPipeline
metric = LogLikelihoodAccMetric(normalization=LogProbPMINorm())
batch = {
"choices_text": ["A", "B"],
# Only conditioned logprobs are possible via current API:
"choices_logprob": [-1.2, -1.5],
# There's no supported field to provide unconditioned logprobs here.
}
pipe = EvaluationPipeline(metrics=[metric])
pipe.add_batch(batch)
pipe.evaluate()
# => AssertionError: unconditioned_logprob must be provided for PMI normalization
```
Attempting a workaround by adding both conditioned and unconditioned values to `choices_logprob` causes:
```
ValueError: choices_text and choices_logprob must have the same length.
```
## Expected behavior
One of the following should be supported:
1. **Documented input schema:**
Provide a clear way to supply unconditioned logprobs per choice.
For example:
```python
"choices_logprob": [-1.2, -1.5], # conditioned
"choices_uncond_logprob": [-0.9, -1.1] # unconditioned
```
2. **Automatic computation:**
If the backend can compute unconditioned logprobs automatically, PMI normalization should trigger that process internally.
3. **Better messaging / removal:**
If PMI normalization is not yet implemented, remove `LogProbPMINorm()` from the normalization options and update docs and error messages accordingly.
## Version info
* **OS:** [Ubuntu 22.04.3 LTS]
* **LightEval:** [0.11.0]
* **Backend:** [vllm,accelerate]
* **Python:** [3.10,3.12]
## Additional context / logs
* Error 1:
```
AssertionError: unconditioned_logprob must be provided for PMI normalization
```
* Error 2:
```
ValueError: choices_text and choices_logprob must have the same length.
```
[pmi.log](https://github.com/user-attachments/files/22598570/pmi.log)
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.