ag-ui-protocol / ag-ui-protocol/ag-ui

[Bug]: TS langgraph converters drop media type + metadata; multimodal round-trip is lossy

未关闭
#2,011 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
bug Integration
主要语言
Python
星标
15.9k
派生
1.4k
平均合并
1 天 17 小时
30 天内合并 PR
163

描述

## Pre-flight

Searched existing issues — this extends #1809 (Python-only, one-directional; see "Relationship to #1809" below), it is not a duplicate.

## Describe the Bug

The TypeScript LangGraph integration (`integrations/langgraph/typescript/src/utils.ts`) loses multimodal metadata and the original media type in both conversion directions, so an attachment cannot survive an AG-UI → LangGraph → AG-UI round-trip:

- **`convertAguiMultimodalToLangchain` (AG-UI → LangChain)** collapses every `image | audio | video | document` part to a flat `{ type: "image_url", image_url: { url } }`, discarding `InputContent.metadata` and the original type.

- **`convertLangchainMultimodalToAgui` (LangChain → AG-UI)** re-emits every `image_url` as `{ type: "image", source: {...} }` — hard-coded to `image`, no metadata. The code even comments: *"LangChain only uses image_url blocks for all media, so we always produce ImageInputContent"* and *"The true media type is not recoverable."*

**Net effect:** after LangGraph echoes a stored message back via `MESSAGES_SNAPSHOT` (live send or reload), an audio/video/document attachment returns as a generic image with no metadata. UIs that render attachment chips from `type`/`metadata` can't tell a PDF from an image, and any caller annotations on `InputContent.metadata` are gone.

## Steps to Reproduce

1. Build a `HumanMessage` whose content includes a `DocumentInputContent` (or audio/video) with `metadata`.
2. Run it through `aguiMessagesToLangChain` → store/echo via LangGraph → `langchainMessagesToAgui`.
3. Result is `type: "image"` with no metadata; original type and metadata are lost.

## Expected Behavior

The original media type and metadata should survive the round-trip. A block that genuinely carries no type info should still fall back to `image` (no regression).

## Proposed Fix

Stash the original AG-UI type as a namespaced key inside the `metadata` object (`__agui_type`) so no extra top-level property is needed on the LangChain block.

**Forward (`convertAguiMultimodalToLangchain`):**
```ts
// before
{ type: "image_url", image_url: { url } }

// after
{ type: "image_url", image_url: { url }, metadata: { ...item.metadata, __agui_type: item.type } }
```

**Reverse (`convertLangchainMultimodalToAgui`):**
```ts
const aguiType = block.metadata?.__agui_type ?? "image";
const cleanMeta = block.metadata
? Object.fromEntries(Object.entries(block.metadata).filter(([k]) => k !== "__agui_type"))
: undefined;
// produce the correct InputContent subtype from aguiType + cleanMeta
```

The `metadata` field already survives the checkpoint JSON round-trip (that's the mechanism #1832 relied on), so `__agui_type` comes along for free. The only edge case is a caller who already has a `__agui_type` key in their metadata — acceptable given how unlikely and easily documented that is.

**Longer term:** emit LangChain v1 standard content blocks (typed `image`/`file`/`video`/`audio`) and carry AG-UI metadata in the block `extras` field. langchain-ai/langgraphjs#1838 was closed with a deprecation plan: the SDK's `Message` types are being replaced by `@langchain/react` primitives that auto-coerce to `@langchain/core` message instances (v1 content block API), with `coerceMessageLikeToMessage` as the interim bridge. So this path is more accessible than previously assumed — but doesn't affect the near-term fix above, which doesn't depend on it.

**Scope:** `src/utils.ts` (two functions + their inline block types) plus regression tests in `src/utils.test.ts` — no public API change, no new deps.

## Relationship to #1809 / PR #1832

`#1809` + merged PR `#1832` fixed only the Python `convert_agui_multimodal_to_langchain`, only the forward direction, and only metadata (it still emits `type: "image_url"`, so the original media type is still dropped, and the reverse converter is untouched). This issue tracks (a) TypeScript parity, (b) the reverse converter, and (c) preserving the original media type so the round-trip is actually lossless.

## Environment

`@ag-ui/langgraph` (TypeScript integration), current main.

贡献指南

打开贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。