langgenius / langgenius/dify

Agent App: Images generated by third-party tools are not displayed in chat — image URLs discarded in _convert_tool_response_to_text

Open
#40,425 2 comments 1 reaction 0 assignees View on GitHub
Dominant language
TypeScript
Stars
156k
Forks
24.6k
Avg merge
22h 9m
Merged PRs (30d)
610

Description

## What happened

When an Agent App (the new Agent backend path, not the legacy Agent Chat) calls a third-party tool that returns an image (e.g. an image generation tool that returns `MessageType.IMAGE_LINK` or `MessageType.IMAGE`), the image is never displayed to the user in the chat interface. The user only sees a text response like "image has been created and sent to user already, you do not need to create it, just tell the user to check it now."

## Root Cause

There are two layers where image data is lost:

### 1. dify-agent SDK side — `_convert_tool_response_to_text` discards image URLs

File: `dify-agent/src/dify_agent/layers/dify_plugin/tools_layer.py`, lines 636-681

```python
def _convert_tool_response_to_text(tool_response: Sequence[DifyPluginToolInvokeMessage]) -> str:
...
elif response.type in {
DifyPluginToolInvokeMessage.MessageType.IMAGE_LINK,
DifyPluginToolInvokeMessage.MessageType.IMAGE,
}:
parts.append(
"image has been created and sent to user already, "
"you do not need to create it, just tell the user to check it now."
)
# ❌ Image URL in message.text is discarded
```

The plugin daemon returns `DifyPluginToolInvokeMessage` with `message.text` containing the image URL, but `_convert_tool_response_to_text` replaces it with a fixed text string. The image URL is lost at this point.

### 2. Dify API side — no compensation mechanism to push images to frontend

The legacy Agent Chat path (`FunctionCallAgentRunner` / `CotAgentRunner`) has a compensation chain that runs **before** converting tool output to text:
- `ToolFileMessageTransformer.transform_tool_invoke_messages()` downloads and stores images as `ToolFile`
- `ToolEngine._create_message_files()` creates `MessageFile` DB records
- The runner publishes `QueueMessageFileEvent` to the SSE stream
- The frontend receives `message_file` SSE events and renders the image

The new Agent App path (`AgentAppRunner` in `api/core/app/apps/agent_app/app_runner.py`) has **no equivalent mechanism**:
- `_AgentProcessRecorder._handle_tool_result_event()` stores tool results as observation text only
- `MessageAgentThought.message_files` is always `""` (empty string)
- `QueueMessageFileEvent` is never imported or published
- `_message_end_to_stream_response()` queries `MessageFile` table → 0 records → `files=[]`
- Frontend `onFile` callback never fires

## Reproduction

1. Create an Agent App (not Agent Chat / not Workflow)
2. Configure a third-party image generation tool (e.g. DALL-E, Stable Diffusion, or any tool that returns `MessageType.IMAGE` / `MessageType.IMAGE_LINK`)
3. Send a message requesting image generation
4. Agent calls the tool, tool successfully generates an image
5. Chat only shows text "image has been created and sent to user already..." — no image is displayed

## Expected Behavior

The image generated by the tool should be displayed in the chat interface, similar to how the legacy Agent Chat path handles tool-generated images.

## Flow Comparison

```
Legacy Agent Chat (images display correctly ✅):
ToolEngine.agent_invoke()
→ ToolFileMessageTransformer: download image, store as ToolFile
→ _create_message_files(): create MessageFile record (type=IMAGE, belongs_to=ASSISTANT)
→ publish QueueMessageFileEvent → SSE: message_file event → frontend renders ✅
→ tool_response_to_str(): "image has been created..." (text for LLM only)
→ QueueMessageEndEvent → message_end.files = [image list] → frontend renders ✅

New Agent App (images lost ❌):
dify-agent SDK: invoke_tool closure
→ _convert_tool_response_to_text(): "image has been created..." ← ❌ URL discarded
→ pydantic-ai ToolReturnPart.content = fixed text (no image)
→ emit_run_succeeded(output="text response")
Dify API: AgentAppRunner._consume_stream()
→ _AgentProcessRecorder: stores observation text only, message_files=""
→ No MessageFile created, No QueueMessageFileEvent published
→ SSE: agent_message (text only), message_end (files=[])
→ Frontend: no image ❌
```

## Suggested Fix

Two approaches:

**Option A: Fix in dify-agent SDK** — Return image information from `_convert_tool_response_to_text` alongside the text, so it can flow through pydantic-ai events to the Dify API side, which then creates `MessageFile` records and publishes `QueueMessageFileEvent`.

**Option B: Add compensation in Dify API** — In `AgentAppRunner._consume_stream()` / `_AgentProcessRecorder._handle_tool_result_event()`, extract file references from tool results, create `MessageFile` records (similar to legacy `ToolEngine._create_message_files`), and publish `QueueMessageFileEvent` to the SSE stream.

## Environment

- Dify version: latest `main` branch (cloned for source analysis)
- Path affected: Agent App (new Agent backend path)
- Key files:
- `dify-agent/src/dify_agent/layers/dify_plugin/tools_layer.py` (lines 636-681)
- `api/core/app/apps/agent_app/app_runner.py` (lines 250-570)
- `api/core/workflow/nodes/agent_v2/agent_node.py` (lines 502-590)
- `api/core/tools/tool_engine.py` (legacy reference, lines 49-157)

Contributor guide

Open the contributing guide

Research direction

Start with dify-agent/src/dify_agent/layers/dify_plugin/tools_layer.py and trace _convert_tool_response_to_text for IMAGE and IMAGE_LINK results. Then read api/core/app/apps/agent_app/app_runner.py, api/core/workflow/nodes/agent_v2/agent_node.py, and the legacy flow in api/core/tools/tool_engine.py. Done means Agent App tool-generated images create message files, publish the expected SSE events, and render in chat without breaking the text response.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai, api, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.