Bug: Response::body_json could not convert from an UTF-8 BOM source file
- Dominant language
- Rust
- Stars
- 1.5k
- Forks
- 128
- PR merge metrics
- No merged PRs in 30d
Description
## Target
- `body_json` function of `Response` struct
-
## Detail
- `body_json` function could not convert from an UTF-8 BOM source file
## Repro
1. Prepare an `UTF-8 BOM` JSON file.; BOM bytes is `0xef 0xbb 0xbf`.
2. Deploy the JSON file and run a httpd.
3. `surf::get` and `body_json` => `SerdeJsonError(Error("expected value", line: 1, column: 1))`
## Workaround
```rust
// It will be the error if response has the BOM.
// let my_struct = my_response.body_json::();
```
1. Raw implementation:
```rust
let my_response_body_string = my_response.body_string();
let my_response_body_string = if my_response_body_string.starts_with("\u{feff}") { &my_response_body_string[3..] } else { &my_response_body_string[..] };
let my_struct: MyStruct = serde_json::from_str(my_response_body_string);
```
2. Or use [`strip_bom`](https://crates.io/crates/strip_bom):
```rust
use strip_bom::StripBom;
let my_response_body_string = my_response.body_string();
let my_response_body_string = my_response_body_string.strip_bom();
let my_struct: MyStruct = serde_json::from_str(my_response_body_string);
```
---
Notes:
1. This error is `SerdeJsonError`, but a character encoding issue might not issue of `serde_json` or `serde` crate. Because, `serde_json` is designed for UTF-8 stream body only, not for a file directly or BOM. Thus, I think the BOM issue is not a `serde_json` or `serde` issue, it is a library user side issue. If you authors think it is not a surf issue, it is a serde_json or serde issue then I'll throw the issue to serde_json or serde repos.
- `serde_json::from_reader` (for an IO stream, ≈ Input UTF-8 character stream)
- `serde_json::from_str` (≈Input UTF-8 character stream).
2. It might be not a `std::string::String` issue because https://github.com/rust-lang/rfcs/issues/2428
3. JSON ([ECMA-404](https://www.ecma-international.org/publications/standards/Ecma-404.htm), [RFC8259](https://tools.ietf.org/html/rfc8259)) is defined the character encoding to use only UTF-8, but the BOM is not defined. Thus, JSON file with UTF-8 BOM is not an invalid in specification. Yes, of course I was think nobody who use UTF-8 BOM to JSON file before today, but a Japanese government institution used it. 🥺
Contributor guide
Assessment
This issue has not been assessed yet.