Agent App: Images generated by third-party tools are not displayed in chat — image URLs discarded in _convert_tool_response_to_text
- Dominant language
- TypeScript
- Stars
- 156k
- Forks
- 24.6k
- Avg merge
- 22h 9m
- Merged PRs (30d)
- 610
Description
## What happened
When an Agent App (the new Agent backend path, not the legacy Agent Chat) calls a third-party tool that returns an image (e.g. an image generation tool that returns `MessageType.IMAGE_LINK` or `MessageType.IMAGE`), the image is never displayed to the user in the chat interface. The user only sees a text response like "image has been created and sent to user already, you do not need to create it, just tell the user to check it now."
## Root Cause
There are two layers where image data is lost:
### 1. dify-agent SDK side — `_convert_tool_response_to_text` discards image URLs
File: `dify-agent/src/dify_agent/layers/dify_plugin/tools_layer.py`, lines 636-681
```python
def _convert_tool_response_to_text(tool_response: Sequence[DifyPluginToolInvokeMessage]) -> str:
...
elif response.type in {
DifyPluginToolInvokeMessage.MessageType.IMAGE_LINK,
DifyPluginToolInvokeMessage.MessageType.IMAGE,
}:
parts.append(
"image has been created and sent to user already, "
"you do not need to create it, just tell the user to check it now."
)
# ❌ Image URL in message.text is discarded
```
The plugin daemon returns `DifyPluginToolInvokeMessage` with `message.text` containing the image URL, but `_convert_tool_response_to_text` replaces it with a fixed text string. The image URL is lost at this point.
### 2. Dify API side — no compensation mechanism to push images to frontend
The legacy Agent Chat path (`FunctionCallAgentRunner` / `CotAgentRunner`) has a compensation chain that runs **before** converting tool output to text:
- `ToolFileMessageTransformer.transform_tool_invoke_messages()` downloads and stores images as `ToolFile`
- `ToolEngine._create_message_files()` creates `MessageFile` DB records
- The runner publishes `QueueMessageFileEvent` to the SSE stream
- The frontend receives `message_file` SSE events and renders the image
The new Agent App path (`AgentAppRunner` in `api/core/app/apps/agent_app/app_runner.py`) has **no equivalent mechanism**:
- `_AgentProcessRecorder._handle_tool_result_event()` stores tool results as observation text only
- `MessageAgentThought.message_files` is always `""` (empty string)
- `QueueMessageFileEvent` is never imported or published
- `_message_end_to_stream_response()` queries `MessageFile` table → 0 records → `files=[]`
- Frontend `onFile` callback never fires
## Reproduction
1. Create an Agent App (not Agent Chat / not Workflow)
2. Configure a third-party image generation tool (e.g. DALL-E, Stable Diffusion, or any tool that returns `MessageType.IMAGE` / `MessageType.IMAGE_LINK`)
3. Send a message requesting image generation
4. Agent calls the tool, tool successfully generates an image
5. Chat only shows text "image has been created and sent to user already..." — no image is displayed
## Expected Behavior
The image generated by the tool should be displayed in the chat interface, similar to how the legacy Agent Chat path handles tool-generated images.
## Flow Comparison
```
Legacy Agent Chat (images display correctly ✅):
ToolEngine.agent_invoke()
→ ToolFileMessageTransformer: download image, store as ToolFile
→ _create_message_files(): create MessageFile record (type=IMAGE, belongs_to=ASSISTANT)
→ publish QueueMessageFileEvent → SSE: message_file event → frontend renders ✅
→ tool_response_to_str(): "image has been created..." (text for LLM only)
→ QueueMessageEndEvent → message_end.files = [image list] → frontend renders ✅
New Agent App (images lost ❌):
dify-agent SDK: invoke_tool closure
→ _convert_tool_response_to_text(): "image has been created..." ← ❌ URL discarded
→ pydantic-ai ToolReturnPart.content = fixed text (no image)
→ emit_run_succeeded(output="text response")
Dify API: AgentAppRunner._consume_stream()
→ _AgentProcessRecorder: stores observation text only, message_files=""
→ No MessageFile created, No QueueMessageFileEvent published
→ SSE: agent_message (text only), message_end (files=[])
→ Frontend: no image ❌
```
## Suggested Fix
Two approaches:
**Option A: Fix in dify-agent SDK** — Return image information from `_convert_tool_response_to_text` alongside the text, so it can flow through pydantic-ai events to the Dify API side, which then creates `MessageFile` records and publishes `QueueMessageFileEvent`.
**Option B: Add compensation in Dify API** — In `AgentAppRunner._consume_stream()` / `_AgentProcessRecorder._handle_tool_result_event()`, extract file references from tool results, create `MessageFile` records (similar to legacy `ToolEngine._create_message_files`), and publish `QueueMessageFileEvent` to the SSE stream.
## Environment
- Dify version: latest `main` branch (cloned for source analysis)
- Path affected: Agent App (new Agent backend path)
- Key files:
- `dify-agent/src/dify_agent/layers/dify_plugin/tools_layer.py` (lines 636-681)
- `api/core/app/apps/agent_app/app_runner.py` (lines 250-570)
- `api/core/workflow/nodes/agent_v2/agent_node.py` (lines 502-590)
- `api/core/tools/tool_engine.py` (legacy reference, lines 49-157)
Contributor guide
Research direction
Start with dify-agent/src/dify_agent/layers/dify_plugin/tools_layer.py and trace _convert_tool_response_to_text for IMAGE and IMAGE_LINK results. Then read api/core/app/apps/agent_app/app_runner.py, api/core/workflow/nodes/agent_v2/agent_node.py, and the legacy flow in api/core/tools/tool_engine.py. Done means Agent App tool-generated images create message files, publish the expected SSE events, and render in chat without breaking the text response.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai, api, backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 48/100