googleapis / googleapis/python-genai

[Feedback/Feature] Dynamic thinking budget floor & explicit sampling controls for multimodal reasoning in Gemini Flash

Open
#2,954 1 comment 0 reactions 1 assignee Claimed by @Venkaiahbabuneelam View on GitHub
priority: p3 type: feature request
Dominant language
Python
Stars
4k
Forks
1k
Avg merge
2d 11h
Merged PRs (30d)
40

Description

### Summary
We have conducted extensive empirical testing comparing `gemini-3.8-flash` across reasoning effort levels (`Medium` vs. `High`) on complex multimodal tasks (fine-grained OCR on compressed video, phonetic audio matching under background noise, and multi-step regulatory decision-making).

### Observed Behavior
1. **Truncated Test-Time Compute in `Medium`:**
- On ambiguous OCR tasks (e.g., distinguishing digits `5` vs. `6` on compressed 240p/720p video feeds or legacy VGA BIOS fonts), `Medium` thinking effort frequently locks in early on statistical priors (e.g., hallucinating version `1.50` instead of `1.60` despite the text being explicitly visible).
- Once a false premise is established in the initial reasoning tokens, the output generation locks onto the hallucinated token with ~99% confidence, bypassing default sampling filters.

2. **Impact of Fixed Sampling (`temperature: 1.0`, `top_p: 0.95`):**
- While `High` reasoning effort completely eliminates these hallucinations due to sufficient compute for cross-frame verification and hypothesis pruning, `Medium` suffers from early budget exhaustion under `temperature: 1.0`.

3. **In `High` Mode:**
- Accuracy reaches ~100% across fine OCR, license plate reconstruction (`2291 KWJ`), noisy audio parsing, and complex DGT traffic regulations.

### Proposed Suggestions for the SDK / Backend:
1. **Dynamic Thinking Floor for Multimodal Inputs:**
- Implement an adaptive minimum token budget for `Medium` when video or dense audio tokens are present in the payload, ensuring at least one full verification/falsification pass.
2. **Explicit SDK & UI Controls for `top_p` and `thinking_budget`:**
- Keep `top_p` and explicit `thinking_budget` integer controls accessible in `types.GenerateContentConfig` and clearly surfaced in AI Studio documentation for power users who need deterministic calibration.
3. **Improved `FinishReason.MALFORMED_RESPONSE` Handling:**
- Enhance client-side retry or fallback handling when reasoning outputs encounter malformed structures during rapid token generation.

### Environment
- **SDK:** `googleapis/python-genai`
- **OS:** Debian 12 (Bookworm) / Linux
- **Model:** `gemini-3.8-flash` / `gemini-3.7-flash`

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.