google-deepmind / google-deepmind/gemma

Bug: Missing f-prefix in error message in _normalize_token() makes stop_tokens debugging impossible

Open
#658 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
5.7k
Forks
1k
Avg merge
10h 33m
Merged PRs (30d)
2

Description

## Summary

The error message in `_normalize_token()` inside `gemma/gm/text/_sampler.py` is missing the `f` prefix, so `{token!r}` is printed **literally** instead of interpolating the actual token value. This makes debugging `stop_tokens` / `forbidden_tokens` misconfigurations nearly impossible.

## Affected Code

**File:** `gemma/gm/text/_sampler.py`, lines 579–581

```python
# Current (broken — NOT an f-string):
raise ValueError(
'Invalid token: {token!r}. `stop_token`s and `forbidden_token`s must'
' map to single token ids in the vocab.'
)
```

## Root Cause

The string literal uses `{token!r}` but there is no `f` prefix, so Python treats it as a plain string. The user sees the unhelpful message:

```
ValueError: Invalid token: {token!r}. `stop_token`s and `forbidden_token`s must map to single token ids in the vocab.
```

Instead of the intended:

```
ValueError: Invalid token: 'hello world'. `stop_token`s and `forbidden_token`s must map to single token ids in the vocab.
```

## Reproduction

```python
from gemma.gm.text._sampler import _normalize_token

class FakeTokenizer:
def encode(self, text):
return [1, 2] # multi-token, will trigger the error

_normalize_token(FakeTokenizer(), 'hello world')
# => ValueError: Invalid token: {token!r}. ... <-- literal, not interpolated
```

## Fix

```python
# Before:
raise ValueError(
'Invalid token: {token!r}. `stop_token`s and `forbidden_token`s must'
' map to single token ids in the vocab.'
)

# After:
raise ValueError(
f'Invalid token: {token!r}. `stop_token`s and `forbidden_token`s must'
' map to single token ids in the vocab.'
)
```

A one-character fix — just add `f` before the opening quote.

## Impact

- **Severity:** Low (no correctness impact — the error is still raised), but **High UX impact** — users who misconfigure `stop_tokens` or `forbidden_tokens` receive a completely useless error message with no information about which token was invalid.
- All platforms affected.

## Environment

| | |
|---|---|
| **Python** | 3.12.13 |
| **Platform** | macOS Apple Silicon (arm64) |
| **gemma** | 4.0.1 (editable install from HEAD) |

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.