google-gemini / google-gemini/gemini-cli

bug: web-fetch grounding citation markers inserted at wrong positions (UTF-16 splice of UTF-8 byte offsets)

Open Beginner friendly
#29,039 0 comments 0 reactions 0 assignees View on GitHub
area/agent status/need-triage
Dominant language
TypeScript
Stars
107k
Forks
14.6k
Avg merge
2d 3h
Merged PRs (30d)
45

Description

## What happened?

`web-fetch` inserts grounding citation markers by splicing a **string array** at `segment.endIndex`, but the Gemini API's grounding segment indices are **UTF-8 byte offsets** — not UTF-16 code-unit positions. For any page whose processed text contains non-ASCII content, markers land at wrong positions; if an offset falls between a surrogate pair, the splice splits it and permanently mojibakes the content.

The sibling implementation for the identical metadata shape, `web-search.ts`, explicitly documents and handles this correctly:

```ts
// Use TextEncoder/TextDecoder since segment indices are UTF-8 byte positions
const encoder = new TextEncoder();
```

Both consume the same `GroundingSupportSegment`, so they cannot both be right.

## Affected code

`packages/core/src/tools/web-fetch.ts:841-846`:

```ts
insertions.sort((a, b) => b.index - a.index);
const responseChars = responseText.split('');
insertions.forEach((insertion) => {
responseChars.splice(insertion.index, 0, insertion.marker);
});
responseText = responseChars.join('');
```

Contrast with the correct approach in `packages/core/src/tools/web-search.ts:145-166` (TextEncoder/TextDecoder byte splicing).

## How can this be reproduced?

1. Fetch any page whose extracted text contains CJK/accented characters or emoji before the first citation point.
2. Compare marker positions against the source text: `[n]` markers appear shifted earlier/later.
3. With emoji in the text: a byte offset landing mid-surrogate-pair produces replacement-character corruption in `llmContent`.

## What did you expect to happen?

Markers inserted at the correct character position regardless of script; no surrogate splitting.

## Suggested direction

Port web-fetch to the same TextEncoder/TextDecoder byte-splice implementation used by web-search (ideally extract it into one shared helper).

---

*Found by source audit on current `main` (commit `5411f113c`); platform-independent. No open issue/PR covering this was found (searched: grounding citation offset UTF).*

Contributor guide

Open the contributing guide

Research direction

Start at packages/core/src/tools/web-fetch.ts:841-846 and compare its insertion logic with the TextEncoder/TextDecoder implementation in packages/core/src/tools/web-search.ts:145-166. Port or share the byte-offset handling so markers stay aligned for non-ASCII text and surrogate pairs are not corrupted; verify the affected web-fetch behavior with relevant tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
tooling
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
88/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.