ag-ui-protocol / ag-ui-protocol/ag-ui

[Bug]: TS langgraph converters drop media type + metadata; multimodal round-trip is lossy

Đang mở
#2,011 0 bình luận 0 reaction 0 người được giao Xem trên GitHub
bug Integration
Ngôn ngữ chính
Python
Star
15.9k
Fork
1.4k
Merge trung bình
1 ngày 17 giờ
Pull request đã merge (30 ngày)
163

Mô tả

## Pre-flight

Searched existing issues — this extends #1809 (Python-only, one-directional; see "Relationship to #1809" below), it is not a duplicate.

## Describe the Bug

The TypeScript LangGraph integration (`integrations/langgraph/typescript/src/utils.ts`) loses multimodal metadata and the original media type in both conversion directions, so an attachment cannot survive an AG-UI → LangGraph → AG-UI round-trip:

- **`convertAguiMultimodalToLangchain` (AG-UI → LangChain)** collapses every `image | audio | video | document` part to a flat `{ type: "image_url", image_url: { url } }`, discarding `InputContent.metadata` and the original type.

- **`convertLangchainMultimodalToAgui` (LangChain → AG-UI)** re-emits every `image_url` as `{ type: "image", source: {...} }` — hard-coded to `image`, no metadata. The code even comments: *"LangChain only uses image_url blocks for all media, so we always produce ImageInputContent"* and *"The true media type is not recoverable."*

**Net effect:** after LangGraph echoes a stored message back via `MESSAGES_SNAPSHOT` (live send or reload), an audio/video/document attachment returns as a generic image with no metadata. UIs that render attachment chips from `type`/`metadata` can't tell a PDF from an image, and any caller annotations on `InputContent.metadata` are gone.

## Steps to Reproduce

1. Build a `HumanMessage` whose content includes a `DocumentInputContent` (or audio/video) with `metadata`.
2. Run it through `aguiMessagesToLangChain` → store/echo via LangGraph → `langchainMessagesToAgui`.
3. Result is `type: "image"` with no metadata; original type and metadata are lost.

## Expected Behavior

The original media type and metadata should survive the round-trip. A block that genuinely carries no type info should still fall back to `image` (no regression).

## Proposed Fix

Stash the original AG-UI type as a namespaced key inside the `metadata` object (`__agui_type`) so no extra top-level property is needed on the LangChain block.

**Forward (`convertAguiMultimodalToLangchain`):**
```ts
// before
{ type: "image_url", image_url: { url } }

// after
{ type: "image_url", image_url: { url }, metadata: { ...item.metadata, __agui_type: item.type } }
```

**Reverse (`convertLangchainMultimodalToAgui`):**
```ts
const aguiType = block.metadata?.__agui_type ?? "image";
const cleanMeta = block.metadata
? Object.fromEntries(Object.entries(block.metadata).filter(([k]) => k !== "__agui_type"))
: undefined;
// produce the correct InputContent subtype from aguiType + cleanMeta
```

The `metadata` field already survives the checkpoint JSON round-trip (that's the mechanism #1832 relied on), so `__agui_type` comes along for free. The only edge case is a caller who already has a `__agui_type` key in their metadata — acceptable given how unlikely and easily documented that is.

**Longer term:** emit LangChain v1 standard content blocks (typed `image`/`file`/`video`/`audio`) and carry AG-UI metadata in the block `extras` field. langchain-ai/langgraphjs#1838 was closed with a deprecation plan: the SDK's `Message` types are being replaced by `@langchain/react` primitives that auto-coerce to `@langchain/core` message instances (v1 content block API), with `coerceMessageLikeToMessage` as the interim bridge. So this path is more accessible than previously assumed — but doesn't affect the near-term fix above, which doesn't depend on it.

**Scope:** `src/utils.ts` (two functions + their inline block types) plus regression tests in `src/utils.test.ts` — no public API change, no new deps.

## Relationship to #1809 / PR #1832

`#1809` + merged PR `#1832` fixed only the Python `convert_agui_multimodal_to_langchain`, only the forward direction, and only metadata (it still emits `type: "image_url"`, so the original media type is still dropped, and the reverse converter is untouched). This issue tracks (a) TypeScript parity, (b) the reverse converter, and (c) preserving the original media type so the round-trip is actually lossless.

## Environment

`@ag-ui/langgraph` (TypeScript integration), current main.

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.