google / google/adk-python

[BUG]: LiteLlm.capabilities hardcodes output_schema_and_tools=True, bypassing the SetModelResponseTool workaround

Aperta
#6,954 4 commenti 0 reazioni 1 assegnatario Rivendicata da @sanketpatil06 Vedi su GitHub
models
Lingua principale
Python
Stelle
21.5k
Fork
4k
Merge medio
1g 14h
PR unite (30g)
37

Descrizione

## Description

`LiteLlm.capabilities` returns `output_schema_and_tools=True` unconditionally, regardless of which model it wraps. ADK trusts that and applies `response_schema` alongside the tool declarations, so `SetModelResponseTool` never engages. On models that cannot honour both constraints the agent loops, calling tools repeatedly instead of producing structured output.

This is the same failure as #3413 and #5054, but on a path none of the existing work covers:

- **#5057** fixed the equivalent gate on the *native* Gemini path. `LiteLlm` bypasses that gate entirely, so merging it would not have helped anyone using ADK through LiteLLM. It was closed as stale because the code moved into `models/_capabilities`, which is the path this report is against.
- **#5091** hardens `SetModelResponseTool` (stronger instructions, forced `tool_config` at round N-1, a 25-round cutoff). All of that applies only once the fallback is active. Here it never activates, because `LiteLlm` reports the capability as supported.

Filing per the request on #5057: *"the repro @wuliang229 asked for is what a version gate needs first... open a new issue if you see this happen again."*

## Environment

- `google-adk` 2.8.0
- `litellm` 1.98.0
- Python 3.13.14
- Model: `openrouter/google/gemini-3.1-flash-lite` (Gemini 3.1 Flash Lite via OpenRouter)

## Reproduction

Needs an OpenRouter key in `OPENROUTER_API_KEY`. Any provider LiteLLM can reach should do; the model family matters more than the provider (see the note on model dependence below).

```python
import asyncio
from pydantic import BaseModel, Field
from google.adk import Agent
from google.adk.models.lite_llm import LiteLlm
from google.adk.runners import InMemoryRunner
from google.genai import types

MODEL = "openrouter/google/gemini-3.1-flash-lite"

class CityTime(BaseModel):
city: str = Field(description="City name")
time: str = Field(description="Local time")

def get_current_time(city: str) -> dict:
"""Returns the current time in a specified city."""
return {"city": city, "time": "10:30 AM"}

agent = Agent(
model=LiteLlm(MODEL),
name="repro",
instruction="Use get_current_time to look up the time, then answer.",
tools=[get_current_time],
output_schema=CityTime,
)

async def main():
runner = InMemoryRunner(agent=agent, app_name="repro")
await runner.session_service.create_session(
app_name="repro", user_id="u", session_id="s")
tool_calls = 0
async for ev in runner.run_async(
user_id="u", session_id="s",
new_message=types.Content(role="user",
parts=[types.Part(text="What time is it in Sydney?")]),
):
if ev.content and ev.content.parts:
for p in ev.content.parts:
if p.function_call:
tool_calls += 1
print(f" tool call {tool_calls}: {p.function_call.name}")
print(f"total tool calls: {tool_calls}")

asyncio.run(main())
```

Observed: `get_current_time` is called repeatedly with identical arguments before the agent settles. Four runs of this exact code gave **4, 12, 18 and 19** tool calls. It does eventually produce valid output, but the count is neither stable nor bounded, and on some runs it did not terminate within several minutes.

Adding `output_schema_and_tools=False` (via the subclass below) makes it **2 function calls, every run**: `get_current_time` plus `set_model_response`.

### Supplementary: confirming the cause without API calls

This inspects the outgoing request and short-circuits before anything is sent, so it needs no key and no credit. Useful for verifying a fix quickly.

```python
import asyncio
from pydantic import BaseModel
from google.adk import Agent
from google.adk.models.lite_llm import LiteLlm
from google.adk.models.base_llm import LlmCapabilities
from google.adk.models.llm_response import LlmResponse
from google.adk.runners import InMemoryRunner
from google.genai import types

MODEL = "openrouter/google/gemini-3.1-flash-lite"

class CityTime(BaseModel):
city: str
time: str

def get_current_time(city: str) -> dict:
"""Returns the current time in a specified city."""
return {"city": city, "time": "10:30 AM"}

class SchemaSafeLiteLlm(LiteLlm):
@property
def capabilities(self) -> LlmCapabilities:
return LlmCapabilities(output_schema_and_tools=False)

def spy(callback_context, llm_request):
cfg = llm_request.config
schema_set = bool(getattr(cfg, "response_schema", None)
or getattr(cfg, "response_json_schema", None))
names = [fd.name
for t in (getattr(cfg, "tools", None) or [])
for fd in (getattr(t, "function_declarations", None) or [])]
print(f" response_schema_set={schema_set} tools_offered={names}")
return LlmResponse(content=types.Content(role="model", parts=[types.Part(text="{}")]))

async def inspect(label, model):
print(f"\n{label} output_schema_and_tools={model.capabilities.output_schema_and_tools}")
agent = Agent(model=model, name="probe", instruction="x",
tools=[get_current_time], output_schema=CityTime,
before_model_callback=spy)
runner = InMemoryRunner(agent=agent, app_name=label)
await runner.session_service.create_session(
app_name=label, user_id="u", session_id="s")
async for _ in runner.run_async(
user_id="u", session_id="s",
new_message=types.Content(role="user", parts=[types.Part(text="time in Sydney?")]),
):
pass

async def main():
await inspect("stock LiteLlm", LiteLlm(MODEL))
await inspect("SchemaSafeLiteLlm", SchemaSafeLiteLlm(MODEL))

asyncio.run(main())
```

Output:

```
stock LiteLlm output_schema_and_tools=True
response_schema_set=True tools_offered=['get_current_time']

SchemaSafeLiteLlm output_schema_and_tools=False
response_schema_set=False tools_offered=['set_model_response', 'get_current_time']
```

## Cost

Each retry resends the whole conversation, so the waste is worse than the call count suggests. Measured for the single question above, via OpenRouter:

| | generations | input tokens | cost |
|---|---|---|---|
| schema only, no tools | 1 | 19 | $0.000071 |
| schema + tools, stock `LiteLlm` | 13 | 5,810 | $0.001848 |

Input tokens grow ~300x against ~26x on cost, so this degrades sharply with a realistic tool payload rather than the toy one used here. The failure is also silent: valid output is eventually returned, so nothing surfaces except the bill.

## Expected behaviour

`LiteLlm` should not assert the capability on behalf of every model it wraps. When the wrapped model cannot pair `response_schema` with tools, ADK's existing `SetModelResponseTool` workaround should engage, as it does on the native path.

With the capability declared `False`, behaviour is correct and stable: **2 function calls per turn** (`get_current_time` plus `set_model_response`), and a second conversation turn still calls the tool rather than answering from history.

## Root cause

`src/google/adk/models/lite_llm.py:3075`

```python
@property
@override
def capabilities(self) -> LlmCapabilities:
# LiteLLM reconciles tools + response_format per provider: providers with
# native support get both passed through, and the rest are converted to a
# json tool call with tool_choice enforcement.
return LlmCapabilities(output_schema_and_tools=True)
```

`src/google/adk/flows/llm_flows/basic.py:137`

```python
if not agent.tools or model.capabilities.output_schema_and_tools:
llm_request.set_output_schema(agent.output_schema)
```

The comment states the reasoning: LiteLLM is expected to reconcile the two per provider. For this provider and model it does not, and both constraints reach the model raw, as the reproduction shows.

By contrast the native path is conservative. `gemini_output_schema_and_tools` requires Vertex AI *and* a Gemini model id, so on other backends it correctly returns `False` and the workaround engages.

## Note on model dependence

The model matters too, consistent with the table in #5054: flash models prefer calling tools over `set_model_response`, while `gemini-3.1-pro-preview` succeeds on the first call. The `gemini-3.1-flash-lite` used here is in the susceptible family. That is an argument for `LiteLlm` being conservative by default rather than asserting support for every wrapped model, since it cannot
know which family it is proxying.

## Possible fixes

1. Default `LiteLlm.capabilities` to `output_schema_and_tools=False`, so the reliable `SetModelResponseTool` path is used unless a user opts out. Safe, at the cost of one extra function call for providers that could have handled it natively.
2. Make it constructor-configurable, e.g. `LiteLlm(model, output_schema_and_tools=False)`, so users can declare it without subclassing.

Deriving it from LiteLLM's own metadata does **not** work. LiteLLM reports both capabilities independently, and has no API for the combination:

```python
litellm.supports_response_schema(model="openrouter/google/gemini-3.1-flash-lite") # True
litellm.supports_function_calling(model="openrouter/google/gemini-3.1-flash-lite") # True
```

Both are True while the combination still loops, so the assumption in the current code comment cannot be rescued by consulting LiteLLM.

Happy to open a PR — combining 1 and 2 (a constructor argument defaulting to `False`) is a small diff, needs no network to test, and makes `LiteLlm` consistent with the conservative native gate. Glad to keep the default at `True` instead if preserving current behaviour matters more.

## Related

- #3413 — output_schema + tools infinite loop
- #5054 — same loop with `output_schema=str`; contains the per-model comparison table
- #5057 — equivalent gate on the native Gemini path; closed as stale after the check moved into `models/_capabilities`. Does not cover the `LiteLlm` path.
- #5091 — hardens `SetModelResponseTool`; helps only once the fallback is active, which it never is here.

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.