ag-ui-protocol / ag-ui-protocol/ag-ui

LangGraph adapter: non-image attachments come back as images (PDF renders as a broken image)

Offen
#2,489 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
Python
Sterne
15.9k
Forks
1.4k
Ø Merge
1 T. 17 Std.
Gemergte PRs (30 T.)
163

Beschreibung

## What happens

Attach a PDF to a message in a LangGraph-backed app. The composer stages it correctly as a document. About two seconds later — when the run echoes the conversation back — it turns into an image, and the UI shows a broken-image placeholder instead of the document. It stays broken across a page reload, because the persisted history is the corrupted form.

Observed in a CopilotKit app, sampling the React props of the attachment renderer through a single run:

```
document | application/pdf | Meridian-Creative-Q2-invoice.pdf <- composer, correct
image | application/pdf | undefined <- 2s later, after the round trip
```

Note the second line is self-contradictory: `type: "image"` with `mimeType: "application/pdf"`. Any client that switches on `type` (CopilotKit's `CopilotChatAttachmentRenderer` does, and so do the Angular/Vue/react-ui equivalents) renders an ``, the decode fails, and the user sees "Failed to load image".

## Why

`integrations/langgraph/python/ag_ui_langgraph/utils.py`.

Outbound, `convert_agui_multimodal_to_langchain` routes **all** media through `image_url`, which is correct and unavoidable — it is the only media block LangChain accepts.

Inbound, `convert_langchain_multimodal_to_agui` rebuilds **every** `image_url` block as `ImageInputContent`. It already parses the real mime type out of the data URL and stores it on the source, then discards the distinction it just recovered:

```python
mime_type = header.split(":")[1].split(";")[0] if ":" in header else "image/png"

agui_content.append(ImageInputContent( # <- always an image
type="image",
source=InputContentDataSource(type="data", value=data, mime_type=mime_type),
))
```

So the modality is destroyed on the return leg. Worth flagging: the outbound docstring reasons that dropping `metadata` loses nothing because "the metadata is never read back on the LangChain->AG-UI return path". That is true of `metadata` specifically, but the *modality* is lost regardless of metadata, because the return path can only construct one class.

## Blast radius

Any non-image attachment through the LangGraph integration: PDFs and other documents, plus audio and video, which also come back as `ImageInputContent` today.

Downstream adapters have started papering over it. This one, for instance, rewrites `image_url` blocks carrying non-image data URLs into LangChain `file` blocks before the model call, purely so the model gets a usable attachment:

```python
def _repair_media_block(block):
"""Turn an `image_url` block carrying a NON-image data URL into a file block."""
```

That fixes what the *model* sees but not what the *client* holds, so the broken chip survives.

## Proposed fix

Read the mime type the function already has, and choose the content class from it: `image/` -> image, `audio/` -> audio, `video/` -> video, anything else -> document. Default the unknown case to document, since a mime type the adapter has never heard of is still not a picture.

Only the `data:` branch needs to change. A bare `https://` URL announces no mime type, so there is nothing to decide on and it should stay an image — the case that already works. Outbound stays untouched, so the model payload is byte-identical and remains spec-compliant for strict OpenAI-compatible providers (#2100).

PR follows, with the round trip covered by tests.

## Not covered by that fix

`InputContent.metadata` is still dropped outbound, so a document's **filename** does not survive the round trip either — clients fall back to showing the mime type. That is legible where a broken image was not, but restoring the filename means carrying it across the LangChain hop, which is the provider-compatibility question #2100 was about. Left as a separate follow-up.

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.