langgenius / langgenius/dify

FC agent runner: detect zero-tool-call terminal answers when a tool call was expected, instead of silently accepting a fabricated response

Open
#40,672 8 comments 1 reaction 0 assignees View on GitHub
💪 enhancement
Dominant language
TypeScript
Stars
156k
Forks
24.6k
Avg merge
22h 9m
Merged PRs (30d)
610

Description

### Self Checks

- [X] I have read the [Contributing Guide]() and [Language Policy]().
- [X] I have searched for existing issues [search for existing issues](), including closed ones.
- [X] I confirm that I am using English to submit this report, otherwise it will be closed.
- [X] Please do not modify this template :) and fill in all the required fields.

### 1\. Is this request related to a challenge you're experiencing? Tell me about your story.

Follow-up to langgenius/dify#40671, where the root cause of a tool-call fabrication bug
was diagnosed down to the runner logic, confirmed by @dosu against the
actual fc_agent_runner.py source.

In the FC agent runner's main loop, function_call_state defaults to False
at the top of each iteration and only flips to True if check_tool_calls()
or check_blocking_tool_calls() detects a tool call in the model's
response. If the model returns plain text instead, the loop exits after
one iteration, and that text becomes final_answer with no validation that
a tool call was actually expected or required.

The signal that this happened is already present — tool_responses stays
empty and function_call_state stays False — it just isn't acted on.
There's currently no way to distinguish "the agent had nothing to do"
from "the agent was supposed to call a tool and didn't."

We hit this in production: an agent configured to delete records via a
tool call sometimes fabricated a "deleted successfully" message with zero
actual tool invocation in the trace. We've had to permanently disable the
affected tools rather than risk this — which is exactly the kind of gap
this request is meant to close.

**What I'd like to see:** a way to mark specific tools (or agent
configurations generally) as requiring a tool call when invoked in a
context that expects one, and detect when a terminal answer is produced
with zero tool invocations in that context. When detected, expose this as
a distinct error or retry path, rather than silently returning the
model's fabricated text as a successful answer. A structural fix at the
runner level would let us (and presumably others) re-enable delete/write-
action tools with confidence, rather than relying entirely on the
underlying model's tool-calling reliability.

### 2\. Additional context or comments

Alternatives already tried, none of which address the underlying gap:

* System prompt enforcement ("You MUST call the tool") — reduces but does
not eliminate the failure, since it's a model-level probabilistic
behavior, not something promptable away.
* Renaming tool parameters to avoid vocabulary overlap — currently
testing this separately; will share results in langgenius/dify#40671 if useful.
* Switching agent strategy to ReAct — a different mechanism, but a real
behavioral trade-off, not evaluated yet for our use case.

Full diagnosis and six rounds of trial data (showing the failure is
probabilistic and correlated with semantic overlap between tool-adjacent
vocabulary and processed data, not fixable via prompting alone) are in
langgenius/dify#40671.

### 3\. Can you help us with this feature?

- [ ] I am interested in contributing to this feature.

Contributor guide

Open the contributing guide

Research direction

Start with fc_agent_runner.py and the main loop around function_call_state, check_tool_calls(), check_blocking_tool_calls(), and tool_responses; review the diagnosis in issue #40671 for context. Define how required-tool configurations identify an expected call, then ensure zero-tool terminal answers take a distinct error or retry path instead of being accepted as final responses.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai, backend-api-design
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.