"Error: could not decode body as UTF-8" on some websites.
- Dominant language
- Rust
- Stars
- 1.5k
- Forks
- 128
- PR merge metrics
- No merged PRs in 30d
Description
On some websites, e.g. http://orgonelab.org, surf fails with the following error:
> Error: could not decode body as UTF-8
It is also sometimes reported for other encodings.
Firefox, curl and ureq (a blocking Rust client) work fine.
28348 websites out of the top million from [Feb 3 Tranco list](https://tranco-list.eu/list/3G6L) are affected.
Tested using [this code](https://github.com/Shnatsel/rust-http-clients-smoke-test/blob/f206362f2e81521bbefb84007cdd25242f6db590/surf-smoke-test/src/main.rs). Test tool output from all affected websites: [surf-could-not-decode-as.tar.gz](https://github.com/http-rs/surf/files/6054894/surf-could-not-decode-as.tar.gz)
I've only tested the async-h1 backend; I don't know if the other backends are affected.
Contributor guide
Research direction
Start by reproducing the failure with the linked smoke-test code against orgonelab.org, then inspect the async-h1 backend, which is the only backend tested in the report. Compare the behavior with the affected-site output and check whether other backends are involved. Done means affected responses no longer fail with decoding errors while existing response handling still works.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100