Fenced code blocks don't render incrementally during streaming

Fenced code blocks don't render incrementally during streaming

Description

When streaming AI responses token-by-token, fenced code blocks (triple backtick) appear to render all at once rather than incrementally as the markdown string grows. Text outside code blocks streams smoothly with visible character-by-character rendering, but the moment a code fence opens, the content inside appears to buffer until the block is complete, then dumps visually.

Environment

• streamdown: 2.5.0
• @streamdown/code: 1.1.1
• React 18
• Using <Streamdown animated isAnimating plugins={{ cjk, code, math, mermaid }}>

Reproduction

1. Pass a growing markdown string to <Streamdown> that includes a fenced code block
2. The string updates token-by-token (verified via console.log — length increments: 4, 8, 24, 26, 34, 58... smoothly up to 860+)
3. Prose text before and after the code fence streams beautifully with the animated caret
4. The code block content appears all at once or in large chunks

Minimal usage:

<Streamdown animated isAnimating plugins={{ cjk, code, math, mermaid }}>
  {streamingMarkdownString}
</Streamdown>

Where streamingMarkdownString grows incrementally and contains:

Some text before...

```typescript
// This content appears to dump all at once
function example() {
  return "hello";
}
```

What we investigated

1. Transport is not the issue

Console logging confirms the raw string updates smoothly, token-by-token. The buffering is in the rendering layer, not the data transport.

2. parseMarkdownIntoBlocks uses marked's Lexer.lex()

parseMarkdownIntoBlocks calls Lexer.lex(markdown, { gfm: true }) and iterates tokens. When the lexer encounters a fenced code block, it emits a single code token only after the closing fence is found. While the fence is open (no closing backticks yet), the raw content likely falls into a different token type or gets merged differently.

Key question: Does marked's Lexer.lex() emit a code-type token for a partial fence (no closing backticks), or does it fall back to a paragraph token? If it treats unclosed fences as paragraphs, the block would render as text during streaming and then suddenly swap to a code block when the closing fence arrives — which is exactly what we observe.

3. isIncomplete infrastructure exists

The Streamdown component checks for unclosed code fences (via hasIncompleteCodeFence) on the last block and passes isIncomplete={true} to the Block component. The useIsCodeFenceIncomplete hook exposes this. However, if marked's lexer doesn't produce a code-type token for the unclosed fence, the isIncomplete flag may not trigger the code block rendering path.

4. chatbot.ai-sdk.dev achieves smooth code block streaming

The reference chatbot at chatbot.ai-sdk.dev (which uses Streamdown via MessageResponse) renders code blocks incrementally during streaming. This suggests either a configuration difference or that AI SDK's useChat delivers data in a way that interacts with Streamdown differently than a custom WebSocket stream.

5. parseIncompleteMarkdown prop

We noticed this prop defaults to false. Does remend processing affect how unclosed code fences are rendered during streaming? Does enabling parseIncompleteMarkdown={true} change the behavior?

Our setup (not using AI SDK)

We're NOT using AI SDK's useChat. We have a custom WebSocket streaming pipeline that delivers text tokens from a container running Claude CLI. The growing markdown string is accumulated in a Zustand store and passed as children to Streamdown. The string IS raw markdown with fences.

Questions

1. Does parseMarkdownIntoBlocks produce incremental block updates for unclosed code fences?
2. Is there a prop or configuration we're missing to enable incremental code block rendering?
3. Does chatbot.ai-sdk.dev use any additional configuration beyond what's in its open-source MessageResponse wrapper?
4. Could remend preprocessing be the key — does it close unclosed fences before the lexer sees them?

Reproduce the issue with a growing markdown string passed to Streamdown, then read parseMarkdownIntoBlocks and marked's Lexer.lex behavior for unclosed fences. Trace hasIncompleteCodeFence, Block, useIsCodeFenceIncomplete, and parseIncompleteMarkdown to see whether partial code tokens reach the code rendering path. Done means fenced code content renders incrementally during streaming without regressing prose rendering.