http-rs / http-rs/surf

Bug: Response::body_json could not convert from an UTF-8 BOM source file

Open
#228 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
1.5k
Forks
128
PR merge metrics
No merged PRs in 30d

Description

## Target

- `body_json` function of `Response` struct
-

## Detail

- `body_json` function could not convert from an UTF-8 BOM source file

## Repro

1. Prepare an `UTF-8 BOM` JSON file.; BOM bytes is `0xef 0xbb 0xbf`.
2. Deploy the JSON file and run a httpd.
3. `surf::get` and `body_json` => `SerdeJsonError(Error("expected value", line: 1, column: 1))`

## Workaround

```rust
// It will be the error if response has the BOM.
// let my_struct = my_response.body_json::();
```

1. Raw implementation:

```rust
let my_response_body_string = my_response.body_string();
let my_response_body_string = if my_response_body_string.starts_with("\u{feff}") { &my_response_body_string[3..] } else { &my_response_body_string[..] };
let my_struct: MyStruct = serde_json::from_str(my_response_body_string);
```

2. Or use [`strip_bom`](https://crates.io/crates/strip_bom):

```rust
use strip_bom::StripBom;

let my_response_body_string = my_response.body_string();
let my_response_body_string = my_response_body_string.strip_bom();
let my_struct: MyStruct = serde_json::from_str(my_response_body_string);
```

---
Notes:
1. This error is `SerdeJsonError`, but a character encoding issue might not issue of `serde_json` or `serde` crate. Because, `serde_json` is designed for UTF-8 stream body only, not for a file directly or BOM. Thus, I think the BOM issue is not a `serde_json` or `serde` issue, it is a library user side issue. If you authors think it is not a surf issue, it is a serde_json or serde issue then I'll throw the issue to serde_json or serde repos.
- `serde_json::from_reader` (for an IO stream, ≈ Input UTF-8 character stream)
- `serde_json::from_str` (≈Input UTF-8 character stream).
2. It might be not a `std::string::String` issue because https://github.com/rust-lang/rfcs/issues/2428
3. JSON ([ECMA-404](https://www.ecma-international.org/publications/standards/Ecma-404.htm), [RFC8259](https://tools.ietf.org/html/rfc8259)) is defined the character encoding to use only UTF-8, but the BOM is not defined. Thus, JSON file with UTF-8 BOM is not an invalid in specification. Yes, of course I was think nobody who use UTF-8 BOM to JSON file before today, but a Japanese government institution used it. 🥺

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.