huggingface / huggingface/evaluate

Perplexity metric fails with GPT-2 tokenizer

Open
#766 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.5k
Forks
341
PR merge metrics
No merged PRs in 30d

Description

### Description

While investigating the perplexity metric, I found that it currently fails when used with GPT-2.

### Reproduction

```python
import evaluate

perplexity = evaluate.load("perplexity", module_type="metric")

perplexity.compute(
predictions=["Hello world."],
model_id="gpt2",
)
```

### Error 1

On my setup, the metric fails with:

```text
AttributeError: GPT2Tokenizer has no attribute special_tokens_map_extended
```

The error originates from:

```python
existing_special_tokens = list(tokenizer.special_tokens_map_extended.values())
```

in `metrics/perplexity/perplexity.py`.

### Error 2

After bypassing that code path, the metric fails with:

```text
ValueError: Asking to pad but the tokenizer does not have a padding token.
```

The tokenizer is called with:

```python
padding=True
```

but GPT-2 does not define a padding token by default.

### Environment

* evaluate: 0.4.7.dev0
* transformers: 5.12.1
* datasets: 5.0.0
* torch: 2.12.0+cu130

### Notes

GPT-2 is used throughout the perplexity metric examples and documentation, so I would expect the metric to work with GPT-2 out of the box.

I'd be happy to investigate further and submit a PR with a fix.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.