google-deepmind / google-deepmind/gemma

Error fix

Open
#629 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
5.7k
Forks
1k
Avg merge
10h 33m
Merged PRs (30d)
2

Description

# --- FIX FOR gemma/gm/text/_tokenizer.py ---

class Gemma4Tokenizer(Tokenizer):
"""Tokenizer for Gemma 4 models."""

# ADDITION: Masking multimodal and internal reasoning tokens
FORBIDDEN_TOKENS = (
special_tokens.IMAGE_PLACEHOLDER,
special_tokens.START_OF_IMAGE,
special_tokens.END_OF_IMAGE,
special_tokens.AUDIO_PLACEHOLDER,
special_tokens.START_OF_AUDIO,
special_tokens.END_OF_AUDIO,
special_tokens.THINKING_START,
special_tokens.THINKING_END,
)

def __init__(self, ...):
# existing init logic

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.