google-deepmind / google-deepmind/gemma
Bug: Broken error messages in token normalization—`{token!r}` shown literally instead of actual token value
- Dominant language
- Python
- Stars
- 5.7k
- Forks
- 1k
- Avg merge
- 10h 33m
- Merged PRs (30d)
- 2
Description
## Description
In `_normalize_token`, when a `stop_token` or `forbidden_token` string maps to multiple token IDs, the code raises a `ValueError`. The error message uses `{token!r}` in a regular string instead of an f-string, so users see the literal `{token!r}` instead of the actual invalid token value. This makes it harder to debug misconfigured stop/forbidden tokens.
Example error shown:
```text
ValueError: Invalid token: {token!r}. `stop_token`s and `forbidden_token`s must map to single token ids in the vocab.
```
Instead of:
```text
ValueError: Invalid token: 'hello world'. `stop_token`s and `forbidden_token`s must map to single token ids in the vocab.
```
## Locations
1. **`gemma/gm/text/_sampler.py`**
2. **`gemma/research/t5gemma/sampling.py`**
## Steps to reproduce
1. Create a tokenizer (e.g. `Tokenizer.from_version(3)`).
2. Call `_normalize_token(tokenizer, "hello world")` (or any string that tokenizes to more than one token).
3. The `ValueError` message will contain the literal `{token!r}` instead of the actual token string.
Contributor guide
Assessment
This issue has not been assessed yet.