google-gemini / google-gemini/gemini-cli

bug(web-fetch): hardcoded UTF-8 decoding garbles non-UTF-8 pages (charset=gbk, iso-8859-1, shift_jis, etc.)

Open
#27,980 2 comments 0 reactions 0 assignees View on GitHub
area/agent status/need-triage
Dominant language
TypeScript
Stars
107k
Forks
14.6k
Avg merge
2d 3h
Merged PRs (30d)
45

Description

## What happened?

The `web-fetch` tool always decodes response bodies using `bodyBuffer.toString('utf8')`, ignoring the `charset` parameter in the `Content-Type` header. When fetching pages that use non-UTF-8 encodings (e.g., `charset=gbk`, `charset=iso-8859-1`, `charset=shift_jis`), the returned content is garbled/mojibake.

**Affected code locations** in `packages/core/src/tools/web-fetch.ts`:

```typescript
// line 324 - fallback path
const rawContent = bodyBuffer.toString('utf8');

// line 662 - error response path
let rawResponseText = bodyBuffer.toString('utf8');

// line 692 - text/plain, application/json path
let text = bodyBuffer.toString('utf8');

// line 703 - text/html path
const html = bodyBuffer.toString('utf8');

// line 741 - unknown content type fallback
let text = bodyBuffer.toString('utf8');
```

All five locations hardcode `'utf8'` as the encoding.

## What did you expect to happen?

The tool should parse the `charset` parameter from the `Content-Type` header and use it to decode the response body. For example:

```typescript
function getCharset(contentType: string): string {
const match = contentType.match(/charset=([^\s;]+)/i);
return match?.[1] ?? 'utf-8';
}

const charset = getCharset(contentType);
const rawContent = bodyBuffer.toString(charset as BufferEncoding);
```

If the charset is not recognized by Node.js Buffer, fall back to UTF-8.

## Steps to reproduce

1. Use web-fetch to fetch a page served with `Content-Type: text/html; charset=gbk` (common on Chinese websites)
2. Observe that the returned content contains garbled characters instead of readable Chinese text

## Client information

Client Information

Running on Windows 11, gemini-cli latest stable.

```console
> /about
```

## Additional context

This is particularly impactful for non-Latin web content (Chinese, Japanese, Korean, Arabic, etc.) where legacy encodings like GBK, Shift_JIS, EUC-KR are still widely used on older websites. The `html-to-text` library used for HTML conversion also receives garbled input, compounding the issue.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.