crewAIInc / crewAIInc/crewAI

Gemini 2.0 Flash Thinking window is wrong (1M, not 32k) and Bedrock provider resolves Claude 2.1 to 100k

Open
#7,441 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
58.8k
Forks
8.5k
Avg merge
1d 15h
Merged PRs (30d)
109

Description

Description

Two context-window values in the provider tables are wrong or unreachable:

  1. gemini-2.0-flash-thinking is a 1M-token model, not 32k. Both LLM_CONTEXT_WINDOW_SIZES ("gemini-2.0-flash-thinking-exp-01-21": 32768) and the Gemini native provider ("gemini-2.0-flash-thinking": 32768) carry the 32,768 figure, but that belonged to the December exp-1219 snapshot. The exp-01-21 refresh raised the window to 1,048,576 tokens (see Google's long-context documentation and coverage of the January upgrade). With LLM(model="gemini-2.0-flash-thinking-exp-01-21", is_litellm=True) the litellm path currently returns 32768 × ratio.
  2. The Gemini native provider's 32k entry is also unreachable. "gemini-2.0-flash-thinking": 32768 sits after "gemini-2.0-flash": 1048576 in a first-match startswith loop, so gemini-2.0-flash-thinking-exp-01-21 resolves through the plain gemini-2.0-flash prefix and returns 1M. The two paths disagree with each other.
  3. The Bedrock provider table has no anthropic.claude-v2:1 entry, so Claude 2.1 resolves through the anthropic.claude-v2 prefix (Claude 2.0) and returns 100k instead of the 200k window llm.py's shared table already assigns to the :1 model id.
Steps to Reproduce
from crewai import LLM

# litellm path: returns 32768 * 0.85, expected 1048576 * 0.85
LLM(model="gemini-2.0-flash-thinking-exp-01-21", is_litellm=True).get_context_window_size()

# native gemini path: returns 1048576 * 0.85 (accidentally correct, table entry dead)
LLM(model="google/gemini-2.0-flash-thinking-exp-01-21").get_context_window_size()

# native bedrock path: returns 100000 * 0.85, expected 200000 * 0.85
LLM(model="bedrock/anthropic.claude-v2:1").get_context_window_size()
Expected behavior

All three paths return the model's official window. Related to #7436 / #7437 (the lookup-order issue in the shared table); this issue is about the stale values and the missing Bedrock entry.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the provider tables named in the issue: LLM_CONTEXT_WINDOW_SIZES in the shared llm.py path, the Gemini native provider table, and the Bedrock provider table. Run the three supplied LLM.get_context_window_size() reproductions to confirm the lookup results. Done means Gemini Thinking resolves to 1,048,576 in both paths and Bedrock anthropic.claude-v2:1 resolves to 200,000, with the reported ratios applied.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.