http-rs / http-rs/surf

"Error: could not decode body as UTF-8" on some websites.

Open
#285 8 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
1.5k
Forks
128
PR merge metrics
No merged PRs in 30d

Description

On some websites, e.g. http://orgonelab.org, surf fails with the following error:

> Error: could not decode body as UTF-8

It is also sometimes reported for other encodings.

Firefox, curl and ureq (a blocking Rust client) work fine.

28348 websites out of the top million from [Feb 3 Tranco list](https://tranco-list.eu/list/3G6L) are affected.

Tested using [this code](https://github.com/Shnatsel/rust-http-clients-smoke-test/blob/f206362f2e81521bbefb84007cdd25242f6db590/surf-smoke-test/src/main.rs). Test tool output from all affected websites: [surf-could-not-decode-as.tar.gz](https://github.com/http-rs/surf/files/6054894/surf-could-not-decode-as.tar.gz)

I've only tested the async-h1 backend; I don't know if the other backends are affected.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the failure with the linked smoke-test code against orgonelab.org, then inspect the async-h1 backend, which is the only backend tested in the report. Compare the behavior with the affected-site output and check whether other backends are involved. Done means affected responses no longer fail with decoding errors while existing response handling still works.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.