nlohmann / nlohmann/json

from_cbor()/from_msgpack() do not validate UTF-8 in text strings at decode time (only dump() does)

Open
#5,529 6 comments 0 reactions 0 assignees View on GitHub

A pull request for this has already been merged.

  • #5531 by @nlohmann — merged
aspect: binary formats kind: bug solution: proposed fix state: please discuss
Dominant language
C++
Stars
50.6k
Forks
7.5k
Avg merge
4d 17h
Merged PRs (30d)
58

Description

Description

from_cbor() (and, by the same code path, from_msgpack()) decodes a CBOR/MessagePack text string by copying its raw bytes into a string_t without validating that they are well-formed UTF-8. RFC 8949 §3.1 requires CBOR major type 3 (text string) to contain "a UTF-8 string", and rejecting ill-formed UTF-8 there is expected decoder behavior — but this library defers that check to dump(), so from_cbor() itself neither throws nor returns a discarded value for invalid UTF-8 payloads. The invalid bytes only surface as an error later, if and when the resulting value is serialized back to JSON text.

Reproduction
#include <nlohmann/json.hpp>
#include <iostream>
using json = nlohmann::json;

int main() {
    // CBOR: 0x62 = text string, length 2; payload 0xC0 0xAE is not valid UTF-8
    std::vector<uint8_t> raw = {0x62, 0xc0, 0xae};

    auto got = json::from_cbor(raw, /*strict*/true, /*allow_exceptions*/false);
    std::cout << "is_discarded: " << got.is_discarded() << "\n";   // false
    std::cout << "type: "        << got.type_name()     << "\n";   // string

    got.dump(); // throws json.exception.type_error.316:
                // "invalid UTF-8 byte at index 0: 0xC0"
}

So from_cbor() reports success (a normal, non-discarded string value) for input the CBOR spec requires to reject; the error only appears later and only if the caller happens to call dump() (or another UTF-8-sensitive operation) on the resulting value.

Traced to binary_reader::get_string(), which forwards straight to get_bytes() — a raw byte copy with no UTF-8 check. Contrast with the text/JSON lexer, which does validate UTF-8 during scanning (scan_string() in lexer.hpp), and with dump()'s own validation (type_error.316).

Why this seems worth a look
  • docs/features/binary_formats/bson.md documents comparable BSON leniency explicitly, with its own "Lenient BSON input handling" admonition and an escape hatch ("validate it separately before passing it to from_bson()"). I could not find an equivalent note for CBOR/MessagePack on the CBOR docs page or the from_cbor API docs — so this specific deferred-validation behavior currently isn't documented anywhere I could find.
  • It means from_cbor(..., /*allow_exceptions=*/false) — the pattern used specifically to avoid exceptions and get a discarded sentinel on bad input instead — does not actually catch this category of malformed input; the exception can still surface later, from an unrelated call (dump()), if the caller isn't also validating UTF-8 there.
Suggested fix / alternatives

Either (a) validate UTF-8 in get_string()/get_bytes() at CBOR/MessagePack decode time, consistent with the text-JSON lexer, or (b) document the deferred-validation behavior the way BSON's leniency is documented, so it's a known, intentional tradeoff rather than a surprise.

Context

Found while investigating a third-party compliance report (#5525) — most of that report's findings turned out to be a bug in the reporting tool itself, but this particular case survived verification against the current develop header (commit 3bfe2b6da7393af5cfd68c44f58a1058e60499e2).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in include/nlohmann/detail/input/binary_reader.hpp at binary_reader::get_string() and get_bytes(), then compare the UTF-8 checks in lexer.hpp's scan_string(). Reproduce the CBOR example and inspect the related CBOR and MessagePack decoding paths; done when malformed text is rejected during decoding or the deferred-validation behavior is explicitly documented, with coverage for the chosen behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.