Idea: canonical / deterministic serialization (RFC 8785 JCS)

Status: brainstorming, not a plan

This is an idea to move forward on, not a sanctioned TODO or a commitment. It is a hypothesis to investigate, measure, and discuss. Treat this as a starting point for discussion.

Motivation

Signing, hashing, content-addressing, reproducible builds, and merkle/blockchain use cases need two serializations of equal documents to be byte-identical. There is currently no way to get that without post-processing the output. RFC 8785 (JSON Canonicalization Scheme, JCS) is the relevant standard.

Of the net-new ideas, this one has the best effort-to-impact ratio: it's a self-contained serializer variant with no core/ABI changes.

What JCS mandates, and where it rubs against dump()

1. Object keys sorted by UTF-16 code unit. The default object_t is std::map, sorted by std::string::operator< — i.e. UTF-8 byte order. For BMP characters that matches UTF-16 order, but for characters ≥ U+10000 it does not (UTF-8 byte order sorts by code point; UTF-16 sorts surrogate pairs, so U+E000–U+FFFF sort after supplementary characters). Canonical mode therefore cannot rely on the container's ordering and must sort keys itself with a UTF-16-code-unit comparator. ordered_json (insertion order) needs the same treatment.

2. Number formatting = ECMAScript Number::toString. Our dtoa already produces shortest round-trip output (≈90% there), but JCS pins the exact rules: plain notation for 1e-6 ≤ |x| < 1e21, exponential outside, specific e+/e- formatting, integers up to 2^53, no trailing .0. This needs an audit against the spec's reference, not an assumption of compatibility.

3. Only finite numbers; UTF-8 output; minimal escaping. NaN/Inf must be a hard error in canonical mode (we currently emit null), and escaping must be the minimal JSON set.

Sketch

A serializer variant, not a tree transform — e.g. dump_canonical() or dump(..., serialization_policy::rfc8785). During traversal it sorts each object's keys into a temporary std::vector<const_iterator> with the UTF-16 comparator and swaps in the ECMAScript number formatter. It never mutates the document, so there's no ABI impact.

std::string canonical = j.dump_canonical();   // illustrative name

Implementation notes

• Lives entirely in the serializer (include/nlohmann/detail/output/serializer.hpp); no changes to basic_json storage.
• UTF-16-code-unit key comparator: compare by transcoding UTF-8 → UTF-16 code units (or an equivalent code-unit-order comparison directly on UTF-8) — this is the easy-to-get-wrong part that passes casual tests but fails above the BMP.
• ECMAScript number formatting: verify the exponent thresholds and integer handling against the RFC 8785 reference vectors; reject non-finite values with a clear error.
• Ship with the official JCS test vectors (RFC 8785 appendix / reference implementation test suite).

Open questions

• API shape: a dedicated method vs. a flag/policy on dump(); how it composes with indent (canonical implies no insignificant whitespace, so indent must be ignored or rejected).
• Where to draw the line vs. full RFC 8785 (e.g. Unicode normalization is explicitly not required by JCS — confirm we don't over-implement).
• Behavior for binary values (not representable in JCS) — error.

Dependencies / interactions

• Deliberately overrides the lossless-number idea: raw number tokens must be re-canonicalized (parse → ECMAScript format), not echoed.
• Independent of the borrowed-view and arena ideas.

Start with include/nlohmann/detail/output/serializer.hpp and RFC 8785, then audit the existing dtoa behavior against the RFC reference vectors. Investigate UTF-16 key ordering, ECMAScript number formatting, non-finite and binary-value errors, and the API and indent behavior. Done means an agreed serializer design validated by the official JCS test vectors.