Python docstring bodies highlight as code in the diff panel because partial patches are tokenized without grammar context

What happened

The diff view syntax highlighting is not correct. A python file, mainly in comment, lines following become green. Block comment did not render green.

Screenshot of the affected diff is attached at the end of this issue.

In a Python file shown in the desktop app's diff panel, the body of a triple-quoted docstring is highlighted as Python source. Prose words inside it (and, in, is, not, If True, escape, are) get keyword colours and bool gets a type colour, and the colouring shifts again after the closing """. A single-line # comment in the same view highlights correctly.

Diagnosis

The diff panel feeds Shiki a synthetic file assembled from the patch's own lines, with no grammar state carried in, so any construct opened above the patch's first line is invisible to the tokenizer.

The chain, at v0.0.40:

1. apps/web/src/lib/diffRendering.ts:119 parses a unified patch with parsePatchFiles. That yields FileDiffMetadata with isPartial: true, which per @pierre/diffs types.d.ts:263-292 means deletionLines/additionLines hold only the context and changed lines present in the patch, not the whole file. compactPartialHunkOffsets (diffRendering.ts:84) keeps collapsedBefore so the gaps render as "N unmodified lines" separators, but those lines are absent from the data.
2. @pierre/diffs@1.3.0-beta.10, dist/utils/renderDiffWithHighlighter.js, groups those lines into buckets and calls renderTwoFiles, which concatenates a bucket's lines into a contents string and passes it to highlighter.codeToHast. Neither Shiki's grammarState nor grammarContextCode is supplied, so every call starts from the initial state.
3. shouldGroupAll = !forcePlainText && !diff.isPartial is false for a partial patch, so getBucketForHunk keys buckets by hunkIndex. This is not where a fix lives: grouping all hunks into one bucket would still concatenate the lines either side of an omitted gap, so the docstring opener would still be missing. Fixing it means supplying the missing context, either full file contents or grammarContextCode.

A Python docstring opened before the patch's first line therefore tokenizes as code. The same applies to any multi-line construct: block strings, template literals, heredocs, embedded languages.

There is one path out of the partial state. components/FileDiff.js:390 expandHunk calls loadFilesIfNecessary, which runs loadDiffFiles and then hydratePartialDiff, setting isPartial = false so the next render tokenizes whole file contents. Both renderers/DiffHunksRenderer.js:1091 and components/VirtualizedFileDiff.js:1020 gate that on hasFileLoader, and apps/web/src/components/DiffPanel.tsx:315-334 returns undefined for the loader unless the view is the active thread's git branch or working-tree diff with selectedTurnId === null. The user reports the wrong colours in every diff view they use, and in none of them were the "N unmodified lines" bands expandable, so no hydration ran anywhere and the colours stayed wrong throughout.

A library upgrade does not fix it. @pierre/diffs 1.4.2, the current latest, carries the same renderDiffWithHighlighter logic; the only changes since beta.10 are an appendItems helper and a default-theme tweak. No grammar-state threading was added.

Steps to reproduce

1. In a Python file, write a function whose docstring body runs longer than the diff's context window, say 15 lines between the opening """ and the closing """.
2. Change one line inside the docstring body, at least four lines below the opening """.
3. Open the diff panel on that change.
4. Expected: the whole docstring body renders in the string colour.
5. Actual: words inside the docstring body are coloured as Python keywords and types, and the colouring changes again at the closing """.
6. The reporter sees this in every diff view, with no expandable "N unmodified lines" bands in any of them, so no hydration ever runs. Where DiffPanel.tsx:315 does supply loadDiffFiles, expanding a band should hydrate to full contents and correct the colours; that bands were never expandable here suggests the loader is absent more widely than selectedTurnId !== null alone accounts for, which is worth checking alongside the tokenizer fix.

Version

0.0.40

Environment

Linux x64 (7.1.1-76070101-generic), Node v24.10.0. Server CLI launched via bunx t3; the UI is the desktop app against a local server on 127.0.0.1:3773.

Evidence

# @pierre/diffs@1.3.0-beta.10 dist/utils/renderDiffWithHighlighter.js
const shouldGroupAll = !forcePlainText && !diff.isPartial;
...
function getBucketForHunk(hunkIndex) {
  const index = shouldGroupAll ? 0 : hunkIndex;
  ...
}

# ... and in renderTwoFiles, in the same file
return getLineNodes(highlighter.codeToHast(cleanLastNewline(additionFile.contents), hastConfig));

# @pierre/diffs@1.3.0-beta.10 dist/types.d.ts, on FileDiffMetadata.isPartial
# "When true, `deletionLines`/`additionLines` contain only the lines present
#  in the patch and hunk expansion is unavailable."

# apps/web/src/lib/diffRendering.ts:119
const parsedPatches = parsePatchFiles(
  normalizedPatch,
  buildPatchCacheKey(normalizedPatch, cacheScope),
);

No server log or trace entries relate to this; the fault is entirely in the renderer.

Related issues

Not a duplicate of #11196 (incremental highlighting), which changes ChatMarkdown only, merged to the legend-perf/incremental-markdown branch after v0.0.40, and states it does not touch the diff viewers. #10830 is theme import, #10822 is progressive diff loading; neither covers grammar state. No existing issue matches.

Fix applied or workaround

Nothing was changed on the machine, and no workaround was available in the view where this was hit. Where a diff view does supply loadDiffFiles, expanding a collapsed context band hydrates the diff to full file contents and restores correct highlighting.

Filed by

claude (opus-5) via t3 triage

Start in apps/web/src/lib/diffRendering.ts around parsePatchFiles and compactPartialHunkOffsets, then trace partial rendering through @pierre/diffs and renderers/DiffHunksRenderer.js. Check the loader path in components/FileDiff.js and apps/web/src/components/DiffPanel.tsx, including why context bands are unavailable. Done means multiline Python docstrings and similar constructs retain correct highlighting in partial diffs, while hydrated diffs remain correct.