serde-rs / serde-rs/json

Deserializing a flattened struct containing a `ByteBuf` fails to deserialize with invalid UTF-8.

Open
#855 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
5.6k
Forks
670
PR merge metrics
No merged PRs in 30d

Description

As per the comment for deserialize_bytes, it is expected that you can deserialize a non-UTF-8 string into a ByteBuf without failure. However, with flattened structures, deserialize_map is called in place of deserialize_struct (see https://github.com/serde-rs/serde/issues/1529), meaning members are deserialized with deserialize_any bypassing deserialize_bytes. deserialize_any assumes values surrounded by quotation marks are valid UTF-8 strings and returns an error otherwise.

An example follows

use serde::{Deserialize, Serialize};

fn main() {
    let a_success: A = serde_json::from_slice(b"{\"b\": {\"buf\": \"\xe5\x00\xe5\"}}").unwrap();
    println!("A Success: {:?}", a_success);

    let a_flat_success: AFlat = serde_json::from_slice(b"{\"buf\": \"abc\"}").unwrap();
    println!("A Flat Success: {:?}", a_flat_success);
    
    let a_flat_failure: AFlat = serde_json::from_slice(b"{\"buf\": \"\xe5\x00\xe5\"}").unwrap();
}

#[derive(Debug, Deserialize, Serialize)]
struct A {
    b: B,
}

#[derive(Debug, Deserialize, Serialize)]
struct AFlat {
    #[serde(flatten)]
    b: B,
}

#[derive(Debug, Deserialize, Serialize)]
struct B {
    buf: serde_bytes::ByteBuf,
}

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the example with A, AFlat, and serde_bytes::ByteBuf, then trace how flattened fields move through deserialize_map and deserialize_any instead of deserialize_struct and deserialize_bytes. Compare the behavior with the existing deserialize_bytes expectation; done means the flattened non-UTF-8 ByteBuf example deserializes successfully without regressing the non-flattened case.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.