FC agent runner: detect zero-tool-call terminal answers when a tool call was expected, instead of silently accepting a fabricated response
- Dominant language
- TypeScript
- Stars
- 156k
- Forks
- 24.6k
- Avg merge
- 22h 9m
- Merged PRs (30d)
- 610
Description
### Self Checks
- [X] I have read the [Contributing Guide]() and [Language Policy]().
- [X] I have searched for existing issues [search for existing issues](), including closed ones.
- [X] I confirm that I am using English to submit this report, otherwise it will be closed.
- [X] Please do not modify this template :) and fill in all the required fields.
### 1\. Is this request related to a challenge you're experiencing? Tell me about your story.
Follow-up to langgenius/dify#40671, where the root cause of a tool-call fabrication bug
was diagnosed down to the runner logic, confirmed by @dosu against the
actual fc_agent_runner.py source.
In the FC agent runner's main loop, function_call_state defaults to False
at the top of each iteration and only flips to True if check_tool_calls()
or check_blocking_tool_calls() detects a tool call in the model's
response. If the model returns plain text instead, the loop exits after
one iteration, and that text becomes final_answer with no validation that
a tool call was actually expected or required.
The signal that this happened is already present — tool_responses stays
empty and function_call_state stays False — it just isn't acted on.
There's currently no way to distinguish "the agent had nothing to do"
from "the agent was supposed to call a tool and didn't."
We hit this in production: an agent configured to delete records via a
tool call sometimes fabricated a "deleted successfully" message with zero
actual tool invocation in the trace. We've had to permanently disable the
affected tools rather than risk this — which is exactly the kind of gap
this request is meant to close.
**What I'd like to see:** a way to mark specific tools (or agent
configurations generally) as requiring a tool call when invoked in a
context that expects one, and detect when a terminal answer is produced
with zero tool invocations in that context. When detected, expose this as
a distinct error or retry path, rather than silently returning the
model's fabricated text as a successful answer. A structural fix at the
runner level would let us (and presumably others) re-enable delete/write-
action tools with confidence, rather than relying entirely on the
underlying model's tool-calling reliability.
### 2\. Additional context or comments
Alternatives already tried, none of which address the underlying gap:
* System prompt enforcement ("You MUST call the tool") — reduces but does
not eliminate the failure, since it's a model-level probabilistic
behavior, not something promptable away.
* Renaming tool parameters to avoid vocabulary overlap — currently
testing this separately; will share results in langgenius/dify#40671 if useful.
* Switching agent strategy to ReAct — a different mechanism, but a real
behavioral trade-off, not evaluated yet for our use case.
Full diagnosis and six rounds of trial data (showing the failure is
probabilistic and correlated with semantic overlap between tool-adjacent
vocabulary and processed data, not fixable via prompting alone) are in
langgenius/dify#40671.
### 3\. Can you help us with this feature?
- [ ] I am interested in contributing to this feature.
Contributor guide
Research direction
Start with fc_agent_runner.py and the main loop around function_call_state, check_tool_calls(), check_blocking_tool_calls(), and tool_responses; review the diagnosis in issue #40671 for context. Define how required-tool configurations identify an expected call, then ensure zero-tool terminal answers take a distinct error or retry path instead of being accepted as final responses.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai, backend-api-design
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100