google-gemini / google-gemini/gemini-cli
bug(web-fetch): hardcoded UTF-8 decoding garbles non-UTF-8 pages (charset=gbk, iso-8859-1, shift_jis, etc.)
- Dominant language
- TypeScript
- Stars
- 107k
- Forks
- 14.6k
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 45
Description
## What happened?
The `web-fetch` tool always decodes response bodies using `bodyBuffer.toString('utf8')`, ignoring the `charset` parameter in the `Content-Type` header. When fetching pages that use non-UTF-8 encodings (e.g., `charset=gbk`, `charset=iso-8859-1`, `charset=shift_jis`), the returned content is garbled/mojibake.
**Affected code locations** in `packages/core/src/tools/web-fetch.ts`:
```typescript
// line 324 - fallback path
const rawContent = bodyBuffer.toString('utf8');
// line 662 - error response path
let rawResponseText = bodyBuffer.toString('utf8');
// line 692 - text/plain, application/json path
let text = bodyBuffer.toString('utf8');
// line 703 - text/html path
const html = bodyBuffer.toString('utf8');
// line 741 - unknown content type fallback
let text = bodyBuffer.toString('utf8');
```
All five locations hardcode `'utf8'` as the encoding.
## What did you expect to happen?
The tool should parse the `charset` parameter from the `Content-Type` header and use it to decode the response body. For example:
```typescript
function getCharset(contentType: string): string {
const match = contentType.match(/charset=([^\s;]+)/i);
return match?.[1] ?? 'utf-8';
}
const charset = getCharset(contentType);
const rawContent = bodyBuffer.toString(charset as BufferEncoding);
```
If the charset is not recognized by Node.js Buffer, fall back to UTF-8.
## Steps to reproduce
1. Use web-fetch to fetch a page served with `Content-Type: text/html; charset=gbk` (common on Chinese websites)
2. Observe that the returned content contains garbled characters instead of readable Chinese text
## Client information
Client Information
Running on Windows 11, gemini-cli latest stable.
```console
> /about
```
## Additional context
This is particularly impactful for non-Latin web content (Chinese, Japanese, Korean, Arabic, etc.) where legacy encodings like GBK, Shift_JIS, EUC-KR are still widely used on older websites. The `html-to-text` library used for HTML conversion also receives garbled input, compounding the issue.
Contributor guide
Assessment
This issue has not been assessed yet.