google-gemini / google-gemini/gemini-cli
bug: web-fetch grounding citation markers inserted at wrong positions (UTF-16 splice of UTF-8 byte offsets)
- Dominant language
- TypeScript
- Stars
- 107k
- Forks
- 14.6k
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 45
Description
## What happened?
`web-fetch` inserts grounding citation markers by splicing a **string array** at `segment.endIndex`, but the Gemini API's grounding segment indices are **UTF-8 byte offsets** — not UTF-16 code-unit positions. For any page whose processed text contains non-ASCII content, markers land at wrong positions; if an offset falls between a surrogate pair, the splice splits it and permanently mojibakes the content.
The sibling implementation for the identical metadata shape, `web-search.ts`, explicitly documents and handles this correctly:
```ts
// Use TextEncoder/TextDecoder since segment indices are UTF-8 byte positions
const encoder = new TextEncoder();
```
Both consume the same `GroundingSupportSegment`, so they cannot both be right.
## Affected code
`packages/core/src/tools/web-fetch.ts:841-846`:
```ts
insertions.sort((a, b) => b.index - a.index);
const responseChars = responseText.split('');
insertions.forEach((insertion) => {
responseChars.splice(insertion.index, 0, insertion.marker);
});
responseText = responseChars.join('');
```
Contrast with the correct approach in `packages/core/src/tools/web-search.ts:145-166` (TextEncoder/TextDecoder byte splicing).
## How can this be reproduced?
1. Fetch any page whose extracted text contains CJK/accented characters or emoji before the first citation point.
2. Compare marker positions against the source text: `[n]` markers appear shifted earlier/later.
3. With emoji in the text: a byte offset landing mid-surrogate-pair produces replacement-character corruption in `llmContent`.
## What did you expect to happen?
Markers inserted at the correct character position regardless of script; no surrogate splitting.
## Suggested direction
Port web-fetch to the same TextEncoder/TextDecoder byte-splice implementation used by web-search (ideally extract it into one shared helper).
---
*Found by source audit on current `main` (commit `5411f113c`); platform-independent. No open issue/PR covering this was found (searched: grounding citation offset UTF).*
Contributor guide
Research direction
Start at packages/core/src/tools/web-fetch.ts:841-846 and compare its insertion logic with the TextEncoder/TextDecoder implementation in packages/core/src/tools/web-search.ts:145-166. Port or share the byte-offset handling so markers stay aligned for non-ASCII text and surrogate pairs are not corrupted; verify the affected web-fetch behavior with relevant tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- tooling
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 88/100