googleapis / googleapis/python-genai

max_output_tokens caps thinking + output tokens combined, causing infinite hangs without it

Open
#2,062 4 comments 3 reactions 1 assignee Claimed by @Venkaiahbabuneelam View on GitHub
api: gemini-api type: bug
Dominant language
Python
Stars
4k
Forks
1k
Avg merge
2d 11h
Merged PRs (30d)
40

Description

## Summary

`max_output_tokens` acts as a **combined budget** for thinking tokens + output tokens on Gemini 3 models, contrary to the [documentation](https://ai.google.dev/gemini-api/docs/thinking) which states they are separate. When `max_output_tokens` is not set, the model can enter an infinite thinking loop and hang indefinitely.

## Reproduction

**Model:** `gemini-3-flash-preview`

**Input:** A 380-character Old Assyrian (Akkadian) transliteration with a simple prompt: "Translate the following Old Assyrian (Akkadian) transliteration to English. Return ONLY the English translation, nothing else."

**Expected output:** ~118 tokens of English translation.

### Test 1: Varying `max_output_tokens`

```python
from google import genai
from google.genai import types

client = genai.Client()

for max_tokens in [1024, 8192, 16384, 32768]:
response = client.models.generate_content(
model="gemini-3-flash-preview",
contents="Translate the following Old Assyrian (Akkadian) transliteration to English. "
"Return ONLY the English translation, nothing else.\n\n"
"um-ma en-um-a-šur-ma a-na en-na-sú-in ù DINGIR-ba-ni qí-bi-ma ...",
config=types.GenerateContentConfig(
temperature=0.0,
max_output_tokens=max_tokens,
thinking_config=types.ThinkingConfig(include_thoughts=True),
),
)
print(f"max_output_tokens={max_tokens}")
print(f" thinking: {response.usage_metadata.thoughts_token_count}")
print(f" output: {response.usage_metadata.candidates_token_count}")
```

**Results:**

| `max_output_tokens` | Thinking tokens | Output tokens | Thinking + Output | Finish reason |
|---|---|---|---|---|
| 1,024 | ~906 | truncated | 1,024 | MAX_TOKENS |
| 8,192 | 7,862 | 118 | 7,980 | STOP |
| 16,384 | 15,726 | 118 | 15,844 | STOP |
| 32,768 | 31,455 | 118 | 31,573 | STOP |
| **not set** | **∞** | **never reached** | **∞** | **hangs forever** |

Thinking tokens scale linearly with `max_output_tokens`, consuming ~96% of the budget. The actual output (118 tokens) is identical in every case. The model produces the same translation regardless of how much thinking budget it gets.

### Test 2: Varying `thinking_level` (with `max_output_tokens=16384`)

```python
for level in ['MINIMAL', 'LOW', 'MEDIUM', 'HIGH']:
response = client.models.generate_content(
model="gemini-3-flash-preview",
contents=same_prompt,
config=types.GenerateContentConfig(
temperature=0.0,
max_output_tokens=16384,
thinking_config=types.ThinkingConfig(
thinking_level=level,
include_thoughts=True,
),
),
)
```

| `thinking_level` | Thinking tokens | Output tokens |
|---|---|---|
| MINIMAL | 0 | 125 |
| LOW | 1,377 | 120 |
| MEDIUM | 2,520 | 127 |
| HIGH | 15,726 | 118 |

`thinking_level` does control thinking depth. But at HIGH (the default for Gemini 3), thinking still expands to fill the `max_output_tokens` budget.

### Test 3: No `max_output_tokens` → infinite hang

Without `max_output_tokens`, the API call **hangs indefinitely** (tested for 20+ minutes before killing the process). This happened both inside a `ThreadPoolExecutor` and in a single-threaded call. The same input translates instantly with `max_output_tokens` set, or with `thinking_level=MINIMAL/LOW/MEDIUM`.

We also observed this with `gemini-3-pro-preview` in an agentic setting: the model produced **65,006 output tokens** (526 thinking tokens) in a single call over 7.7 minutes, filling the entire default output limit with text that simulated tool calls instead of actually invoking them.

## Problems

### 1. Documentation mismatch

The [thinking documentation](https://ai.google.dev/gemini-api/docs/thinking) states:

> "When thinking is turned on, response pricing is the sum of output tokens and thinking tokens."

This implies they are tracked and budgeted separately. In practice, `max_output_tokens` acts as a combined cap for both.

### 2. Infinite hang with no `max_output_tokens`

When `max_output_tokens` is not set (which is the default), certain inputs cause the model to think indefinitely. There is no server-side timeout or default cap to prevent this. This is particularly dangerous in agentic/automated pipelines where calls are made programmatically.

### 3. Thinking scales to fill budget with no quality gain

The model produces identical output (same translation, same 118 tokens) whether it thinks for 906, 7,862, 15,726, or 31,455 tokens. The thinking expands to fill ~96% of whatever `max_output_tokens` budget is provided. This wastes tokens and increases latency/cost without improving output quality.

### 4. `max_output_tokens=1024` truncates output, not thinking

At low `max_output_tokens` values, thinking consumes nearly the entire budget, leaving insufficient room for the actual output. The output gets truncated mid-sentence rather than the thinking being capped to preserve output space.

## SDK Analysis

We analyzed the [python-genai SDK source](https://github.com/googleapis/python-genai):

- `max_output_tokens` is passed directly as `maxOutputTokens` in the API request with no transformation (`models.py:1093-1096`)
- `thinking_config` is passed directly as `thinkingConfig` with no transformation (`models.py:1220-1221`)
- There is **no interaction, validation, or cross-referencing** between these two parameters in the SDK
- The combined budgeting behavior is entirely server-side

## Expected Behavior

1. `max_output_tokens` should only cap **output tokens**, not thinking + output combined (matching the docs)
2. There should be a default server-side cap on thinking tokens to prevent infinite hangs
3. Alternatively, clearly document that `max_output_tokens` is a combined budget and recommend always setting it
4. The model should prioritize preserving output completeness over thinking when approaching the token limit

## Environment

- `google-genai` SDK version: latest (pip)
- Models tested: `gemini-3-flash-preview`, `gemini-3-pro-preview`
- Python 3.12

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.