googleapis / googleapis/python-genai

Thinking models are unreliable when `max_output_tokens` set due to them ignoring the thinking budget

Open
#782 53 comments 10 reactions 1 assignee Claimed by @Venkaiahbabuneelam View on GitHub
api: gemini-api api: vertex-ai priority: p2 type: bug
Dominant language
Python
Stars
4k
Forks
1k
Avg merge
2d 11h
Merged PRs (30d)
40

Description

Three different issues make the thinking models (2.5 flash and pro) unreliable:
1. The `thinking_budget` appears to be ignored. We get non-zero (and large) `thoughts_token_count` values regardless of the `thinking_budget` config. This appears to be a [known issue](https://x.com/officiallogank/status/1912986097765789782?s=46&t=HtV6BYzUq3LlVZnyMYQnCg)?
2. A `MAX_TOKENS` finish reason appears to be given if `thoughts_token_count + output_token_count > max_output_tokens`. This is problematic since we don't appear to have proper control over `thoughts_token_count` right now.
3. If `MAX_TOKENS` finish reason is triggered, the response text is empty, making debugging of the above very difficult (we can't get a sense of the size of the output text).

We only started experiencing consistent `MAX_TOKENS` responses sometime last week, so it seems something might have changed recently? Either the default thinking budget, or the way `max_output_tokens` is applied?

The only solution that currently works for us is to set `max_output_tokens=None`, but then the thinking tokens often increase to ~6k even for very simple tasks.

**Edit**: we have only seen this occur when using structured output. I'm can't confirm if it occurs for unstructured output.

#### Environment details

- Programming language: Python
- OS: Ubuntu 22.04
- Language runtime version: 3.10
- Package version: 1.14.0

#### Example

The [docs](https://ai.google.dev/gemini-api/docs/thinking#set-budget) suggest setting `thinking_budget=0` will turn thinking off, however with the following request:
```python
class RelevantDataFields(BaseModel):
fields: List[Literal[tuple(field_names)]]

model_config = GenerateContentConfig(
system_instruction=self.system_instruction,
thinking_config=ThinkingConfig(thinking_budget=0),
max_output_tokens=2000,
temperature=0.,
response_schema=RelevantDataFields.model_json_schema(),
response_mime_type='application/json'
)
pred = await aio.models.generate_content(
model='gemini-2.5-pro-preview-03-25',
contents=prompt,
config=model_config
)
```

the response is:
```python
GenerateContentResponse(candidates=[Candidate(content=Content(parts=[Part(video_metadata=None,
thought=None, code_execution_result=None, executable_code=None, file_data=None, function_call=None,
function_response=None, inline_data=None, text='')], role='model'), citation_metadata=None,
finish_message=None, token_count=None, finish_reason=,
avg_logprobs=None, grounding_metadata=None, index=0, logprobs_result=None, safety_ratings=None)],
create_time=None, response_id=None, model_version='models/gemini-2.5-pro-preview-05-06',
prompt_feedback=None,
usage_metadata=GenerateContentResponseUsageMetadata(cache_tokens_details=None,
cached_content_token_count=None, candidates_token_count=None, candidates_tokens_details=None,
prompt_token_count=1674, prompt_tokens_details=[ModalityTokenCount(modality=, token_count=1674)], thoughts_token_count=2000, tool_use_prompt_token_count=None, tool_use_prompt_tokens_details=None, total_token_count=3674, traffic_type=None), automatic_function_calling_history=[], parsed=None)
```

The `thoughts_token_count=2000` despite setting `thinking_budget=0`, and the model version is also wrong (`05-06` instead of `03-25`).

#### Ideal solution
1. `thinking_budget` should constrain `thoughts_token_count`
2. `MAX_TOKENS` finish reason should only be returned if the number of output tokens is greater than `max_output_tokens`, rather than including the thinking budget (assuming this is the case?).
3. The computed output so far should be returned if `MAX_TOKENS` is reached.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.