langchain-ai / langchain-ai/deepagentsjs

Bug: SummarizationMiddleware internal model call tokens leak into `agent.stream()` output

Open
#629 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
1.6k
Forks
272
Avg merge
1d 13h
Merged PRs (30d)
38

Description

## Bug: SummarizationMiddleware internal model call tokens leak into `agent.stream()` output

### Description

When the summarization middleware triggers, the tokens from its internal `chatModel.invoke()` call **leak into the parent agent's streaming output**. Users see summary generation text (e.g. "Here is a concise summary of the conversation...") mixed into the normal AI response stream. This affects **all stream modes** — `streamMode: 'messages'`, `streamMode: 'updates'`, and `streamEvents` alike. There is no way to distinguish summary output from normal AI output at the stream level — both appear as `msg.type === 'ai'` chunks with the same namespace and no distinguishing metadata.

(Note: `streamEvents` has the same issue — summary model tokens leak into the event stream as well.)

### Root Cause

`createSummarizationMiddleware` reuses the **same model instance** as the parent agent for its internal model call:

```js
// langsmith-wdF8zG42.js line 5773 / 5821
createSummarizationMiddleware({ backend })
// ⚠️ No `model` option passed → falls back to `request.model` (the agent's model instance)
```

This model instance already has **LangGraph streaming callbacks** attached (set up by `agent.stream()`). When `performSummarization` → `summarizeMessages` → `createSummary` calls `chatModel.invoke()`:

```js
// Inside createSummary()
const response = await chatModel.invoke([new HumanMessage({ content: prompt })]);
```

The callbacks inherited from the parent graph context capture every token and emit them through `agent.stream()` as `streamMode: 'messages'` chunks.

This is the same root cause as **LangChain Python PR #34501** (already merged): `ensure_config()` inherits callbacks from `var_child_runnable_config`.

### Relevant Code Path

```
agent.stream(input, { streamMode: ['messages', 'updates'], subgraphs: true })
→ wrap_model_call middleware
→ summarizationMiddleware.wrapModelCall()
→ performSummarization()
→ summarizeMessages()
→ createSummary()
→ chatModel.invoke([summarization prompt]) // ← TOKENS LEAK HERE
→ buildSummaryMessage(summaryText)
→ handler({ messages: [summary_HumanMessage, ...preserved] })
→ agent model generates actual response // ← NORMAL OUTPUT
→ Command({ update: { _summarizationEvent: {...} } })
```

### Observed Behavior

1. **Prompt logs confirm an extra model call**: A `TokenLogger` (`BaseCallbackHandler` attached to the model instance) captures the summarization prompt as a separate invocation
2. **Stream output contains summary text**: The summary AI message appears in `agent.stream()` as `msg.type === 'ai'` chunks
3. **No distinguishing metadata**: Both summary and normal AI chunks have `msg.name === undefined`, identical namespace, and no differentiating `additional_kwargs`. The second element of `data` (LangGraph metadata) is **completely identical** between summary and normal messages — no field can be used to tell them apart.

The user-side stream handling code below shows the problem: summary-leaked AI messages and normal replies go through the same branch with no way to distinguish them:

```ts
for await (const [namespace, mode, data] of stream) {
if (mode === 'messages') {
const msg = Array.isArray(data) ? data[0] : data
// data[1] is LangGraph metadata — identical for summary and normal messages

if (msg.type === 'ai' && !!msg.content) {
// Sub-agent output
if (namespace && namespace.length > 1) {
callbacks.onChunk({ content: msg.content, type: 'think' })
} else {
// ← Both leaked summary and normal reply end up here
// msg.name === undefined (same for both)
// msg.additional_kwargs — no difference
// namespace — same (main agent)
// data[1] metadata — completely identical
callbacks.onChunk({ content: msg.content, type: 'result' })
}
}
}
}
```

4. **After first summarization, it recurs**: `_summarizationEvent` is persisted to checkpoint, causing `getEffectiveMessages()` to prepend the summary `HumanMessage` on every subsequent call. This increases token count, re-triggering summarization repeatedly
5. **Summary HumanMessage IS properly tagged**: `buildSummaryMessage` correctly sets `additional_kwargs: { lc_source: "summarization" }` on the summary HumanMessage, but this only marks the **injected context message**, not the leaked AI output

### Stream data during summarization

```
[messages] AI chunk "Here is a concise summary..." ← LEAKED (no lc_source tag)
[messages] AI chunk "**Main Topics:**..." ← LEAKED
[messages] HumanMessage { lc_source: "summarization" } ← delimiter (properly tagged)
[messages] AI chunk "好呀,想聊什么?" ← normal reply
[messages] AI chunk "..." ← normal reply
[updates] { _summarizationEvent: { cutoffIndex: 14 } }
```

### Related

- **LangChain Python PR #34501** — Same bug, same root cause. Fixed by passing `config={"callbacks": []}` to all internal model calls in `SummarizationMiddleware`, `LLMToolSelectorMiddleware`, `ToolEmulatorMiddleware`. **Already merged.**
- **PR #34763** — Added `metadata: { lc_source: "summarization" }` for UI detection (Python side)

### Proposed Fix

Option A (align with Python fix): Pass `config: { callbacks: [] }` to the internal `chatModel.invoke()` call inside `createSummary()`:

```js
const response = await chatModel.invoke(
[new HumanMessage({ content: prompt })],
{ callbacks: [] } // ← prevent callback inheritance
);
```

Option B (alternative): Allow passing a separate `model` to `createSummarizationMiddleware()`, so it uses a different model instance without the parent graph's callbacks. Currently `createSummarizationMiddleware({ backend })` is called without `model`, causing fallback to `request.model`.

### Environment

- deepagents: `1.10.5`
- `@langchain/langgraph`: `1.4.6`
- `@langchain/core`: `1.2.1`
- LangGraph: with `SqliteSaver` checkpointer
- Model: DeepSeek (ChatDeepSeek), `maxInputTokens` lowered to `10000` via profile override
- Stream config: `{ streamMode: ['messages', 'updates'], subgraphs: true }`

Contributor guide

Open the contributing guide

Research direction

Start at createSummarizationMiddleware and follow performSummarization through summarizeMessages to createSummary, where the internal chatModel.invoke() call is described. Reproduce with agent.stream using messages and updates modes, then verify that summarization text no longer appears as AI chunks while the normal response and summarization update remain.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
ai
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.