google-deepmind / google-deepmind/gemma

Bug: Broken error messages in token normalization—`{token!r}` shown literally instead of actual token value

Open
#579 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
5.7k
Forks
1k
Avg merge
10h 33m
Merged PRs (30d)
2

Description

## Description

In `_normalize_token`, when a `stop_token` or `forbidden_token` string maps to multiple token IDs, the code raises a `ValueError`. The error message uses `{token!r}` in a regular string instead of an f-string, so users see the literal `{token!r}` instead of the actual invalid token value. This makes it harder to debug misconfigured stop/forbidden tokens.

Example error shown:
```text
ValueError: Invalid token: {token!r}. `stop_token`s and `forbidden_token`s must map to single token ids in the vocab.
```

Instead of:
```text
ValueError: Invalid token: 'hello world'. `stop_token`s and `forbidden_token`s must map to single token ids in the vocab.
```

## Locations

1. **`gemma/gm/text/_sampler.py`**

2. **`gemma/research/t5gemma/sampling.py`**

## Steps to reproduce

1. Create a tokenizer (e.g. `Tokenizer.from_version(3)`).
2. Call `_normalize_token(tokenizer, "hello world")` (or any string that tokenizes to more than one token).
3. The `ValueError` message will contain the literal `{token!r}` instead of the actual token string.

Image

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.