google-deepmind / google-deepmind/gemma

Bug: Inconsistent error message in tokenizer validation

Open
#421 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
5.7k
Forks
1k
Avg merge
10h 33m
Merged PRs (30d)
2

Description

**Description**
There is a minor inconsistency between the validation logic and the error message for custom token IDs in the '_add_custom_tokens`'method.

**Location**
- File: 'gemma/gm/text/_tokenizer.py'
- Method: '_add_custom_tokens'

**The Problem**
The code correctly validates that the custom token ID 'i' is within the range of [0, 98]

However, if this condition is met, the ValueError that is raised contains an incorrect message:

raise ValueError(
f'Custom token id {i} for {token!r} is not in [1, 98].'
)
Mismatch between the zero-based indexing used in the validation logic and the one-based counting reflected in the error string.

Image

I have a fix ready and can open a pull request to resolve this.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.