googleapis / googleapis/python-genai
Thinking models are unreliable when `max_output_tokens` set due to them ignoring the thinking budget
- Dominant language
- Python
- Stars
- 4k
- Forks
- 1k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 40
Description
Three different issues make the thinking models (2.5 flash and pro) unreliable:
1. The `thinking_budget` appears to be ignored. We get non-zero (and large) `thoughts_token_count` values regardless of the `thinking_budget` config. This appears to be a [known issue](https://x.com/officiallogank/status/1912986097765789782?s=46&t=HtV6BYzUq3LlVZnyMYQnCg)?
2. A `MAX_TOKENS` finish reason appears to be given if `thoughts_token_count + output_token_count > max_output_tokens`. This is problematic since we don't appear to have proper control over `thoughts_token_count` right now.
3. If `MAX_TOKENS` finish reason is triggered, the response text is empty, making debugging of the above very difficult (we can't get a sense of the size of the output text).
We only started experiencing consistent `MAX_TOKENS` responses sometime last week, so it seems something might have changed recently? Either the default thinking budget, or the way `max_output_tokens` is applied?
The only solution that currently works for us is to set `max_output_tokens=None`, but then the thinking tokens often increase to ~6k even for very simple tasks.
**Edit**: we have only seen this occur when using structured output. I'm can't confirm if it occurs for unstructured output.
#### Environment details
- Programming language: Python
- OS: Ubuntu 22.04
- Language runtime version: 3.10
- Package version: 1.14.0
#### Example
The [docs](https://ai.google.dev/gemini-api/docs/thinking#set-budget) suggest setting `thinking_budget=0` will turn thinking off, however with the following request:
```python
class RelevantDataFields(BaseModel):
fields: List[Literal[tuple(field_names)]]
model_config = GenerateContentConfig(
system_instruction=self.system_instruction,
thinking_config=ThinkingConfig(thinking_budget=0),
max_output_tokens=2000,
temperature=0.,
response_schema=RelevantDataFields.model_json_schema(),
response_mime_type='application/json'
)
pred = await aio.models.generate_content(
model='gemini-2.5-pro-preview-03-25',
contents=prompt,
config=model_config
)
```
the response is:
```python
GenerateContentResponse(candidates=[Candidate(content=Content(parts=[Part(video_metadata=None,
thought=None, code_execution_result=None, executable_code=None, file_data=None, function_call=None,
function_response=None, inline_data=None, text='')], role='model'), citation_metadata=None,
finish_message=None, token_count=None, finish_reason=,
avg_logprobs=None, grounding_metadata=None, index=0, logprobs_result=None, safety_ratings=None)],
create_time=None, response_id=None, model_version='models/gemini-2.5-pro-preview-05-06',
prompt_feedback=None,
usage_metadata=GenerateContentResponseUsageMetadata(cache_tokens_details=None,
cached_content_token_count=None, candidates_token_count=None, candidates_tokens_details=None,
prompt_token_count=1674, prompt_tokens_details=[ModalityTokenCount(modality=, token_count=1674)], thoughts_token_count=2000, tool_use_prompt_token_count=None, tool_use_prompt_tokens_details=None, total_token_count=3674, traffic_type=None), automatic_function_calling_history=[], parsed=None)
```
The `thoughts_token_count=2000` despite setting `thinking_budget=0`, and the model version is also wrong (`05-06` instead of `03-25`).
#### Ideal solution
1. `thinking_budget` should constrain `thoughts_token_count`
2. `MAX_TOKENS` finish reason should only be returned if the number of output tokens is greater than `max_output_tokens`, rather than including the thinking budget (assuming this is the case?).
3. The computed output so far should be returned if `MAX_TOKENS` is reached.
Contributor guide
Assessment
This issue has not been assessed yet.