google-deepmind / google-deepmind/gemma

Feat/auto download tokenizer

Open
#537 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
5.7k
Forks
1k
Avg merge
10h 33m
Merged PRs (30d)
2

Description

#Implement Tokenizer Auto-Download from GCS

## Description
Implements automatic downloading and caching of tokenizer files from GCS (`gs://`) to the local machine. This improves the developer experience by removing the requirement to manually download vocab files from GCS buckets.

## Changes
- **`gemma/gm/utils/_file_cache.py`**:
- Updated `maybe_get_from_cache` to detect `gs://` paths.
- If a `gs://` path is provided and the file is not found in the local cache (`~/.gemma/` by default), it automatically downloads the file using `etils.epath`.
- Automatically creates necessary local directories.
- **`gemma/gm/text/_tokenizer.py`**:
- Removed redundant TODOs regarding auto-downloading, as the functionality is now natively supported by the file cache utility.

## Verification
- Created a verification script `verify_tokenizer_download.py` that mocks `etils.epath` and GCS access.
- Verified that:
1. Local cache hits return the local path immediately.
2. Local cache misses for `gs://` paths trigger a download to the cache directory.
3. Local cache misses for standard local paths still return the original paths for compatibility.

## How to test
Set `GEMMA_CACHE_DIR` to a temporary directory if you want to avoid contaminating your `~/.gemma` directory, then initialize a `Gemma2Tokenizer` or `Gemma3Tokenizer`. The model should download the `.model` file on the first run.

## #528 PR Raised for the issue, lmk if there's any iterations . Love contributing to Google-Deepmind

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.