Workers AI gemma-4: tool_choice 'required' fails open on long contexts; non-stream response discards generated tokens; named-function forcing leaves arguments unconstrained
- Dominant language
- TypeScript
- Stars
- 1.2k
- Forks
- 345
- Avg merge
- 13h 31m
- Merged PRs (30d)
- 1
Description
**Surface:** Workers AI serving for `@cf/google/gemma-4-26b-a4b-it` (REST `/ai/run/...` and the `env.AI` binding; observed identically through AI Gateway). Not a `workers-ai-provider` code issue — filing here because platform-behavior reports for Workers AI/AI Gateway live in this tracker (cf. #547); happy to re-route if there's a better channel.
## Summary
Three related serving-side behaviors around `tool_choice` on `@cf/google/gemma-4-26b-a4b-it`:
1. **`tool_choice: "required"` is advisory under long contexts.** The model schema documents `'required' = must call a tool`, and it does force a call on short prompts. On a real 61-message chat (~9.2k prompt tokens, thinking enabled), the model ignores it and emits free prose. Streaming surfaces that prose to the client as normal `content` deltas.
2. **Non-stream response swallows the generated output when no tool call is parsed.** Replaying the same long-context request with `stream: false` returns:
```json
{
"choices": [{
"finish_reason": "tool_calls",
"message": { "content": "", "reasoning": null, "tool_calls": [] },
"stop_reason": 106
}],
"usage": { "prompt_tokens": 9218, "completion_tokens": 317, ... }
}
```
317 completion tokens were generated and billed, but the response contains no content, no reasoning, an empty `tool_calls` array, and a contradictory `finish_reason: "tool_calls"`. Whatever the right resolution for behavior (1) is, this response shape looks like the tool-call parser dropping the model's actual output on the floor; an error, or returning the text, would both be more truthful.
3. **Named-function forcing constrains the envelope, not the arguments.** With `tool_choice: { "type": "function", "function": { "name": "read_skill_resource" } }` on the same long context, the serving layer does force a call to the named function — but the model can emit free prose *inside* `arguments` (we received the model's conversational reply as the `arguments` string, not JSON matching the declared `parameters` schema). If guided decoding is in play for the function name, extending it to the parameters schema (or documenting that arguments are unconstrained) would help callers.
## Repro sketch
Short context (forcing works — baseline):
```jsonc
POST /accounts/{account}/ai/run/@cf/google/gemma-4-26b-a4b-it
{
"messages": [
{ "role": "system", "content": "You are a warm coach." },
{ "role": "user", "content": "That went really well! How do you think I did?" }
],
"tools": [{ "type": "function", "function": { "name": "read_skill_resource", "description": "Read a bundled resource file.", "parameters": { "type": "object", "properties": { "name": { "type": "string" }, "path": { "type": "string" } }, "required": ["name", "path"] } } }],
"tool_choice": "required",
"max_tokens": 2000,
"chat_template_kwargs": { "enable_thinking": true, "clear_thinking": true }
}
// → finish_reason "tool_calls", proper forced call. Reproduced twice.
```
Long context (forcing fails open): identical request shape, but `messages` is a realistic 61-message coaching conversation (~9.2k prompt tokens) ending in an assistant tool call + tool result. Same single tool, same `tool_choice: "required"`:
- `stream: true` → plain prose streamed as `content` deltas, no tool call.
- `stream: false` → the swallowed-output response shown above (`completion_tokens: 317`, empty message).
- swap in the named-function `tool_choice` → forced call envelope, prose in `arguments`.
I can share exact request IDs / AI Gateway log IDs from our account (Science of People, `efa870654588498c88cc1f2a563323fa`) with Cloudflare folks privately if useful — the failing long-context request is fully captured there, including the `cf-wai-tool-call` IDs.
## Why it matters
The AI SDK ecosystem maps forced tool choice onto these knobs (today `workers-ai-provider` maps `{type:"tool"}` → `"required"`; proposed fix is the named form — filed as #560). Agentic loops built on `prepareStep` tool forcing assume "must call a tool" is a hard constraint; on this model it fails open exactly in the conditions where agents need it most (long multi-turn transcripts), and the non-stream response shape hides that it happened.
Contributor guide
Assessment
This issue has not been assessed yet.