cactus-compute / cactus-compute/cactus

Gemma4 tool-calling emits declaration-format echoes instead of <|tool_call> (function_calls always empty)

Open
#693 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
C++
Stars
6k
Forks
501
Avg merge
1d 18h
Merged PRs (30d)
4

Description

## Summary
For tool prompts, Gemma4 (via cactus) emits the **declaration** format that cactus injects for tool *definitions* (`<|tool>declaration:NAME{...}`, see `gemma_tools.h:69/330`) instead of the **call** format the parser expects (`<|tool_call>...`, `gemma_tools.h:527-530`). As a result `function_calls` is **always empty** — cactus never parses a valid call.

This is **systemic, not specific to multi-tool prompts.**

## Reproduce
```
cactus test --suite llm
```
All three tool cases show `"function_calls": []` and a `<|tool>declaration:...` response:

| Test | Response (truncated) | `function_calls` | suite verdict |
|---|---|---|---|
| TOOL CALL (1 tool) | `<\|tool>declaration:get_weather{...}` | `[]` | PASS* |
| MULTIPLE (2 tools) | `<\|tool>declaration:send_message{...schema echo...}` | `[]` | FAIL |
| TRIPLE (3 tools) | `<\|tool>declaration:send_message<\|"\|> ...` | `[]` | PASS* |

\* The suite's pass/fail only checks whether the right tool **name** appears in the text — it does not require a parsed call. So the 1- and 3-tool cases "pass" despite producing no usable call; the 2-tool case fails that loose check because the echo is more garbled. The actual behavior (declaration echo, no parsed call) is the same in all three.

## Likely root cause (cactus side)
The model is continuing in the `<|tool>declaration:` *definition* syntax cactus injected, rather than switching to the `<|tool_call>` *call* syntax cactus's parser requires. That points to the tool-call prompt/template/instruction in `gemma_tools.h` (how tools are framed and how the model is told to emit calls), not a Gemma model limitation — the model is producing the wrong *kind* of artifact, not a malformed call.

## Not verified
Reference HF Gemma4 with its native tool-call format was not run, so a model/format-fit contribution isn't fully excluded. But the consistent declaration-echo against cactus's **custom** `declaration:` format is the strong signal.

## Scope
Pre-existing; reproduces on `main`/baseline. Unrelated to the KV-compaction / growable-cache work on `kv-compress-keydiff` (found incidentally while running `cactus test --suite llm` during that PR's validation).

Contributor guide

Open the contributing guide

Research direction

Start by running `cactus test --suite llm` and reading the tool framing and parser sections in `gemma_tools.h`, especially lines 69, 330, and 527-530. Trace how Gemma4 receives tool definitions and how responses become `function_calls`; done means the three reported cases produce parsed calls rather than declaration-format echoes, with the suite verifying that behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
ai
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.