google-deepmind / google-deepmind/gemma

[Testing] Missing unit tests for Embedder.encode_vision multimodal path

Open
#544 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
5.7k
Forks
1k
Avg merge
10h 33m
Merged PRs (30d)
2

Description

The `Embedder` module in `gemma/gm/nn/_modules.py` currently implements an `encode_vision` method which acts as the critical "bridge" for multimodal inference. This method projects visual features (e.g., from SigLiP) into the Transformer's unified embedding space using `RMSNorm` and an `Einsum` projection.

Currently, there are no dedicated unit tests for this path, as noted by the TODO at line 74 of `gemma/gm/nn/_modules_test.py`.

**Goal**:
- Implement a robust test suite for `encode_vision`.
- Verify that initializing the `Embedder` with `vision_proj_dim` correctly creates the `mm_input_projection` and `mm_soft_embedding_norm` parameters.
- Ensure that visual tokens are correctly projected to the model's `embed_dim`.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.