ag-ui-protocol / ag-ui-protocol/ag-ui

LangGraph adapter: non-image attachments come back as images (PDF renders as a broken image)

Đang mở
#2,489 0 bình luận 0 reaction 0 người được giao Xem trên GitHub
Ngôn ngữ chính
Python
Star
15.9k
Fork
1.4k
Merge trung bình
1 ngày 17 giờ
Pull request đã merge (30 ngày)
163

Mô tả

## What happens

Attach a PDF to a message in a LangGraph-backed app. The composer stages it correctly as a document. About two seconds later — when the run echoes the conversation back — it turns into an image, and the UI shows a broken-image placeholder instead of the document. It stays broken across a page reload, because the persisted history is the corrupted form.

Observed in a CopilotKit app, sampling the React props of the attachment renderer through a single run:

```
document | application/pdf | Meridian-Creative-Q2-invoice.pdf <- composer, correct
image | application/pdf | undefined <- 2s later, after the round trip
```

Note the second line is self-contradictory: `type: "image"` with `mimeType: "application/pdf"`. Any client that switches on `type` (CopilotKit's `CopilotChatAttachmentRenderer` does, and so do the Angular/Vue/react-ui equivalents) renders an ``, the decode fails, and the user sees "Failed to load image".

## Why

`integrations/langgraph/python/ag_ui_langgraph/utils.py`.

Outbound, `convert_agui_multimodal_to_langchain` routes **all** media through `image_url`, which is correct and unavoidable — it is the only media block LangChain accepts.

Inbound, `convert_langchain_multimodal_to_agui` rebuilds **every** `image_url` block as `ImageInputContent`. It already parses the real mime type out of the data URL and stores it on the source, then discards the distinction it just recovered:

```python
mime_type = header.split(":")[1].split(";")[0] if ":" in header else "image/png"

agui_content.append(ImageInputContent( # <- always an image
type="image",
source=InputContentDataSource(type="data", value=data, mime_type=mime_type),
))
```

So the modality is destroyed on the return leg. Worth flagging: the outbound docstring reasons that dropping `metadata` loses nothing because "the metadata is never read back on the LangChain->AG-UI return path". That is true of `metadata` specifically, but the *modality* is lost regardless of metadata, because the return path can only construct one class.

## Blast radius

Any non-image attachment through the LangGraph integration: PDFs and other documents, plus audio and video, which also come back as `ImageInputContent` today.

Downstream adapters have started papering over it. This one, for instance, rewrites `image_url` blocks carrying non-image data URLs into LangChain `file` blocks before the model call, purely so the model gets a usable attachment:

```python
def _repair_media_block(block):
"""Turn an `image_url` block carrying a NON-image data URL into a file block."""
```

That fixes what the *model* sees but not what the *client* holds, so the broken chip survives.

## Proposed fix

Read the mime type the function already has, and choose the content class from it: `image/` -> image, `audio/` -> audio, `video/` -> video, anything else -> document. Default the unknown case to document, since a mime type the adapter has never heard of is still not a picture.

Only the `data:` branch needs to change. A bare `https://` URL announces no mime type, so there is nothing to decide on and it should stay an image — the case that already works. Outbound stays untouched, so the model payload is byte-identical and remains spec-compliant for strict OpenAI-compatible providers (#2100).

PR follows, with the round trip covered by tests.

## Not covered by that fix

`InputContent.metadata` is still dropped outbound, so a document's **filename** does not survive the round trip either — clients fall back to showing the mime type. That is legible where a broken image was not, but restoring the filename means carrying it across the LangChain hop, which is the provider-compatibility question #2100 was about. Left as a separate follow-up.

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.