crewAIInc / crewAIInc/crewAI

[BUG] Gemini gemini-2.0-flash-thinking returns the wrong context window (891289 instead of 27852)

Open
#7,129 0 comments 0 reactions 0 assignees View on GitHub

A pull request for this has already been merged.

  • #7003 by @a-yeyang — merged
Dominant language
Python
Stars
58.8k
Forks
8.5k
Avg merge
1d 15h
Merged PRs (30d)
109

Description

Description

In GeminiCompletion.get_context_window_size() the lookup table has "gemini-2.0-flash" listed before "gemini-2.0-flash-thinking", and the loop returns on the first match. So the thinking entry is never reached.

# lib/crewai/src/crewai/llms/providers/gemini/completion.py:1356
context_windows = {
    "gemini-3-pro-preview": 1048576,
    "gemini-2.0-flash": 1048576,          # matches first
    "gemini-2.0-flash-thinking": 32768,   # never reached
    "gemini-2.0-flash-lite": 1048576,
    ...
}

for model_prefix, size in context_windows.items():
    if self.model.startswith(model_prefix):
        return int(size * CONTEXT_WINDOW_USAGE_RATIO)

"gemini-2.0-flash-thinking".startswith("gemini-2.0-flash") is True, so the 32768 line is dead code. It has been there since the file was added in v1.0.0 (#3618).

So the model reports a context window of 891289 when the real one is 32768 (27852 after the ratio).

I am not sure how much this affects users in practice, since gemini-2.0-flash-thinking-exp is a preview model. But the value returned is wrong, and the table entry that has the right value can never run.

Steps to Reproduce
  1. Install crewai 1.15.17
  2. Run the code below. No API key or network needed, it is only a table lookup.
  3. Compare the output with int(32768 * 0.85) = 27852
Expected behavior

gemini-2.0-flash-thinking-* should return 27852, which is the value the table already has for it.

In general the lookup should pick the longest matching prefix, so a specific key is not hidden by a shorter key placed above it.

Screenshots/Code snippets
from crewai import LLM

llm = LLM(model="gemini/gemini-2.0-flash-thinking-exp-01-21")
print(llm.get_context_window_size())

Actual:

891289      # this is int(1048576 * 0.85), the gemini-2.0-flash value

Expected:

27852       # int(32768 * 0.85)

One more thing: the same model gives the correct 27852 through crewai/llm.py, because that loop assigns instead of returning and ends on the last match. So the two paths return different values for the same model id.

Additional context

The OpenAI and Azure providers already handle this. They have this comment:

# lib/crewai/src/crewai/llms/providers/openai/completion.py
# Longest prefix first. Always insert new keys in that order so
# startswith prefers gpt-5.6 over gpt-5, gpt-4o-mini over gpt-4o, etc.

The Gemini table does not follow that order. Two ways to fix it:

  1. Move "gemini-2.0-flash-thinking" above "gemini-2.0-flash", same as the other providers do.
  2. Pick the longest matching prefix instead of the first one, so the order does not matter. I would prefer this one, because the table gets edited fairly often (#7012, #7125) and the same mistake can happen again.

I checked the other prefix tables under lib/ with a small script and this is the only entry left whose value cannot be reached.

Happy to send a PR with a regression test if you tell me which of the two fixes you prefer.

Disclosure: I used an AI agent (Claude Code) to help find and check this. I reproduced it locally before filing.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in lib/crewai/src/crewai/llms/providers/gemini/completion.py at GeminiCompletion.get_context_window_size() and compare the matching behavior with crewai/llm.py and the OpenAI provider’s prefix lookup. Run the provided no-network reproduction with the Gemini model identifier. Done means the thinking model returns 27852 and both lookup paths agree, with a regression test covering the prefix collision.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai, backend
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.