google-deepmind / google-deepmind/gemma

Bug: Gemma4Tokenizer missing FORBIDDEN_TOKENS — multimodal placeholder tokens can leak into text-only output

Open
#613 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
5.7k
Forks
1k
Avg merge
10h 33m
Merged PRs (30d)
2

Description

## Description

`Gemma4Tokenizer` does not define `FORBIDDEN_TOKENS`, unlike `Gemma3Tokenizer` and `Gemma3nTokenizer` which both forbid multimodal placeholder tokens from being generated during sampling.

This means when sampling with a Gemma 4 model in text-only mode, the sampler has no restriction on generating raw image/audio placeholder tokens (`<|image|>`, `<|image>`, ``, `<|audio|>`, `<|audio>`, ``), which would produce corrupted output.

## Comparison

```python
# Gemma3Tokenizer (line ~440) — correctly forbids image tokens:
FORBIDDEN_TOKENS = (
special_tokens.START_OF_IMAGE,
special_tokens.END_OF_IMAGE,
)

# Gemma3nTokenizer (line ~465) — same:
FORBIDDEN_TOKENS = (
special_tokens.START_OF_IMAGE,
special_tokens.END_OF_IMAGE,
)

# Gemma4Tokenizer (line ~475) — MISSING, inherits empty tuple from base:
# (no FORBIDDEN_TOKENS defined)
```

## How it's used

In `gemma/gm/text/_sampler.py:501`:
```python
forbidden_tokens += self.tokenizer.FORBIDDEN_TOKENS
```
For Gemma4, this adds nothing, so multimodal tokens are never masked out.

## Proposed Fix

Add `FORBIDDEN_TOKENS` to `Gemma4Tokenizer` covering all multimodal placeholder tokens:

```python
class Gemma4Tokenizer(Tokenizer):
...
FORBIDDEN_TOKENS = (
special_tokens.IMAGE_PLACEHOLDER,
special_tokens.START_OF_IMAGE,
special_tokens.END_OF_IMAGE,
special_tokens.AUDIO_PLACEHOLDER,
special_tokens.START_OF_AUDIO,
special_tokens.END_OF_AUDIO,
)
```

## Location

- File: `gemma/gm/text/_tokenizer.py`
- Class: `Gemma4Tokenizer` (around line 475)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.