ag-ui-protocol / ag-ui-protocol/ag-ui
LangGraph adapter: non-image attachments come back as images (PDF renders as a broken image)
- Vorherrschende Sprache
- Python
- Sterne
- 15.9k
- Forks
- 1.4k
- Ø Merge
- 1 T. 17 Std.
- Gemergte PRs (30 T.)
- 163
Beschreibung
## What happens
Attach a PDF to a message in a LangGraph-backed app. The composer stages it correctly as a document. About two seconds later — when the run echoes the conversation back — it turns into an image, and the UI shows a broken-image placeholder instead of the document. It stays broken across a page reload, because the persisted history is the corrupted form.
Observed in a CopilotKit app, sampling the React props of the attachment renderer through a single run:
```
document | application/pdf | Meridian-Creative-Q2-invoice.pdf <- composer, correct
image | application/pdf | undefined <- 2s later, after the round trip
```
Note the second line is self-contradictory: `type: "image"` with `mimeType: "application/pdf"`. Any client that switches on `type` (CopilotKit's `CopilotChatAttachmentRenderer` does, and so do the Angular/Vue/react-ui equivalents) renders an ``, the decode fails, and the user sees "Failed to load image".
## Why
`integrations/langgraph/python/ag_ui_langgraph/utils.py`.
Outbound, `convert_agui_multimodal_to_langchain` routes **all** media through `image_url`, which is correct and unavoidable — it is the only media block LangChain accepts.
Inbound, `convert_langchain_multimodal_to_agui` rebuilds **every** `image_url` block as `ImageInputContent`. It already parses the real mime type out of the data URL and stores it on the source, then discards the distinction it just recovered:
```python
mime_type = header.split(":")[1].split(";")[0] if ":" in header else "image/png"
agui_content.append(ImageInputContent( # <- always an image
type="image",
source=InputContentDataSource(type="data", value=data, mime_type=mime_type),
))
```
So the modality is destroyed on the return leg. Worth flagging: the outbound docstring reasons that dropping `metadata` loses nothing because "the metadata is never read back on the LangChain->AG-UI return path". That is true of `metadata` specifically, but the *modality* is lost regardless of metadata, because the return path can only construct one class.
## Blast radius
Any non-image attachment through the LangGraph integration: PDFs and other documents, plus audio and video, which also come back as `ImageInputContent` today.
Downstream adapters have started papering over it. This one, for instance, rewrites `image_url` blocks carrying non-image data URLs into LangChain `file` blocks before the model call, purely so the model gets a usable attachment:
```python
def _repair_media_block(block):
"""Turn an `image_url` block carrying a NON-image data URL into a file block."""
```
That fixes what the *model* sees but not what the *client* holds, so the broken chip survives.
## Proposed fix
Read the mime type the function already has, and choose the content class from it: `image/` -> image, `audio/` -> audio, `video/` -> video, anything else -> document. Default the unknown case to document, since a mime type the adapter has never heard of is still not a picture.
Only the `data:` branch needs to change. A bare `https://` URL announces no mime type, so there is nothing to decide on and it should stay an image — the case that already works. Outbound stays untouched, so the model payload is byte-identical and remains spec-compliant for strict OpenAI-compatible providers (#2100).
PR follows, with the round trip covered by tests.
## Not covered by that fix
`InputContent.metadata` is still dropped outbound, so a document's **filename** does not survive the round trip either — clients fall back to showing the mime type. That is legible where a broken image was not, but restoring the filename means carrying it across the LangChain hop, which is the provider-compatibility question #2100 was about. Left as a separate follow-up.
Beitragsleitfaden
Bewertung
Dieses Issue wurde noch nicht bewertet.