google-deepmind / google-deepmind/gemma

Gemma 4 E2B emits parenthetical "word(suffix)" continuations under lower-temperature sampling

Open
#647 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
5.7k
Forks
1k
Avg merge
10h 33m
Merged PRs (30d)
2

Description

## Observation

Gemma 4 E2B (instruction-tuned) appears to have a learned **"word(suffix)" parenthetical continuation style** — the model frequently emits constructions like `offline(text)` / `download(option)` / `feature(name)`, where a noun is immediately followed by a parenthesized clarifier word/phrase, in contexts where this format is not stylistically warranted.

Counter-intuitively, the pattern is **more frequent at lower temperatures**, suggesting it's a high-probability mode of the conditional distribution rather than a noise artifact. At Google's recommended `T=1.0, topP=0.95, topK=64`, the pattern is occasional. At `T=0.4`, it dominates body-text generation in our eval.

## Why this matters

The expectation around lower temperature is "tighter, more deterministic, more on-task." For Gemma 4 E2B, lower temperature instead **amplifies** a specific learned-style attractor. This blocks one of the standard mitigations for a separate Gemma 4 decoding pathology — token repetition / corruption in tool arguments — that we and others (google-deepmind/gemma#622, vllm-project/vllm#40080) have documented.

Concretely: we'd lower temperature for tool-arg segments to reduce the `mayaya`/`dzogogchen`/`202222`-family corruption. But lowering it surfaces the parenthetical pattern in body text. We're stuck at T=1.0.

## Reproduction

- Model: Gemma 4 E2B-it
- Runtime: LiteRT-LM 0.10.2 (also reproducible in vLLM per separate testing)
- Device: Pixel 10a, Mali-G715 GPU
- Sampler: `topK=64, topP=0.95, T=0.4` (degraded), `T=1.0` (acceptable but still occasional)
- Prompt class: general chat / explanatory turns. Most reproducible when the user asks "what is X" for technical X.

## Related observations elsewhere

Closest published thing is google-deepmind/gemma#581 (degenerate embeddings producing fixed attractor states for rare-script tokens), and arXiv 2509.09715 on "symbolic triggers" of hallucination in Gemma. Neither addresses the parenthetical-suffix pattern specifically.

## Ask

1. Acknowledgement that this is a known mode of Gemma 4 E2B's training distribution.
2. Guidance on whether instruction-tuned fine-tunes or future Gemma 4 revisions have addressed it.
3. Any data-side analysis that would help model consumers steer around it (preferred sampling profiles, system-prompt phrasing, etc.).

Happy to provide reproduction prompts + sample outputs.

## References

- google-ai-edge/LiteRT-LM#2202 — full field report on Gemma 4 E2B on-device behaviors
- google-ai-edge/LiteRT-LM#2249 — feature request for `repetitionPenalty` / per-segment sampler
- google-deepmind/gemma#622 — token repetition collapse on larger variants
- google-deepmind/gemma#581 — cuneiform attractor state in Gemma 3

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.