huggingface / huggingface/evaluate
Perplexity metric fails with GPT-2 tokenizer
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 341
- PR merge metrics
- No merged PRs in 30d
Description
### Description
While investigating the perplexity metric, I found that it currently fails when used with GPT-2.
### Reproduction
```python
import evaluate
perplexity = evaluate.load("perplexity", module_type="metric")
perplexity.compute(
predictions=["Hello world."],
model_id="gpt2",
)
```
### Error 1
On my setup, the metric fails with:
```text
AttributeError: GPT2Tokenizer has no attribute special_tokens_map_extended
```
The error originates from:
```python
existing_special_tokens = list(tokenizer.special_tokens_map_extended.values())
```
in `metrics/perplexity/perplexity.py`.
### Error 2
After bypassing that code path, the metric fails with:
```text
ValueError: Asking to pad but the tokenizer does not have a padding token.
```
The tokenizer is called with:
```python
padding=True
```
but GPT-2 does not define a padding token by default.
### Environment
* evaluate: 0.4.7.dev0
* transformers: 5.12.1
* datasets: 5.0.0
* torch: 2.12.0+cu130
### Notes
GPT-2 is used throughout the perplexity metric examples and documentation, so I would expect the metric to work with GPT-2 out of the box.
I'd be happy to investigate further and submit a PR with a fix.
Contributor guide
Assessment
This issue has not been assessed yet.