cloudflare / cloudflare/ai

Workers AI gemma-4: tool_choice 'required' fails open on long contexts; non-stream response discards generated tokens; named-function forcing leaves arguments unconstrained

Open
#561 1 comment 1 reaction 0 assignees View on GitHub
Dominant language
TypeScript
Stars
1.2k
Forks
345
Avg merge
13h 31m
Merged PRs (30d)
1

Description

**Surface:** Workers AI serving for `@cf/google/gemma-4-26b-a4b-it` (REST `/ai/run/...` and the `env.AI` binding; observed identically through AI Gateway). Not a `workers-ai-provider` code issue — filing here because platform-behavior reports for Workers AI/AI Gateway live in this tracker (cf. #547); happy to re-route if there's a better channel.

## Summary

Three related serving-side behaviors around `tool_choice` on `@cf/google/gemma-4-26b-a4b-it`:

1. **`tool_choice: "required"` is advisory under long contexts.** The model schema documents `'required' = must call a tool`, and it does force a call on short prompts. On a real 61-message chat (~9.2k prompt tokens, thinking enabled), the model ignores it and emits free prose. Streaming surfaces that prose to the client as normal `content` deltas.

2. **Non-stream response swallows the generated output when no tool call is parsed.** Replaying the same long-context request with `stream: false` returns:

```json
{
"choices": [{
"finish_reason": "tool_calls",
"message": { "content": "", "reasoning": null, "tool_calls": [] },
"stop_reason": 106
}],
"usage": { "prompt_tokens": 9218, "completion_tokens": 317, ... }
}
```

317 completion tokens were generated and billed, but the response contains no content, no reasoning, an empty `tool_calls` array, and a contradictory `finish_reason: "tool_calls"`. Whatever the right resolution for behavior (1) is, this response shape looks like the tool-call parser dropping the model's actual output on the floor; an error, or returning the text, would both be more truthful.

3. **Named-function forcing constrains the envelope, not the arguments.** With `tool_choice: { "type": "function", "function": { "name": "read_skill_resource" } }` on the same long context, the serving layer does force a call to the named function — but the model can emit free prose *inside* `arguments` (we received the model's conversational reply as the `arguments` string, not JSON matching the declared `parameters` schema). If guided decoding is in play for the function name, extending it to the parameters schema (or documenting that arguments are unconstrained) would help callers.

## Repro sketch

Short context (forcing works — baseline):

```jsonc
POST /accounts/{account}/ai/run/@cf/google/gemma-4-26b-a4b-it
{
"messages": [
{ "role": "system", "content": "You are a warm coach." },
{ "role": "user", "content": "That went really well! How do you think I did?" }
],
"tools": [{ "type": "function", "function": { "name": "read_skill_resource", "description": "Read a bundled resource file.", "parameters": { "type": "object", "properties": { "name": { "type": "string" }, "path": { "type": "string" } }, "required": ["name", "path"] } } }],
"tool_choice": "required",
"max_tokens": 2000,
"chat_template_kwargs": { "enable_thinking": true, "clear_thinking": true }
}
// → finish_reason "tool_calls", proper forced call. Reproduced twice.
```

Long context (forcing fails open): identical request shape, but `messages` is a realistic 61-message coaching conversation (~9.2k prompt tokens) ending in an assistant tool call + tool result. Same single tool, same `tool_choice: "required"`:

- `stream: true` → plain prose streamed as `content` deltas, no tool call.
- `stream: false` → the swallowed-output response shown above (`completion_tokens: 317`, empty message).
- swap in the named-function `tool_choice` → forced call envelope, prose in `arguments`.

I can share exact request IDs / AI Gateway log IDs from our account (Science of People, `efa870654588498c88cc1f2a563323fa`) with Cloudflare folks privately if useful — the failing long-context request is fully captured there, including the `cf-wai-tool-call` IDs.

## Why it matters

The AI SDK ecosystem maps forced tool choice onto these knobs (today `workers-ai-provider` maps `{type:"tool"}` → `"required"`; proposed fix is the named form — filed as #560). Agentic loops built on `prepareStep` tool forcing assume "must call a tool" is a hard constraint; on this model it fails open exactly in the conditions where agents need it most (long multi-turn transcripts), and the non-stream response shape hides that it happened.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.