Unclear requirements of where canonical JSON must be enforced

Link to problem area:

Servers MUST strictly enforce the JSON format specified in the appendices. This translates to a 400 M_BAD_JSON error on most endpoints, or discarding of events over federation. For example, the Federation API’s /send endpoint would discard the event whereas the Client Server API’s /send/{eventType} endpoint would return a M_BAD_JSON error.

— https://spec.matrix.org/v1.11/rooms/v11/#canonical-json

Issue

This is phrasing is very hard to understand and has had multiple interpretations such as:

• JSON over the wire must be in canonical form, except for an undefined set of endpoints
  • @ity:itycodes.org (paraphrased)
• It's unclear what layer this enforcement is occurring at, e.g. on the data over the wire or on data before it is signed/verified, and that most endpoints return that error but other endpoints return a different error (or fail silently)
  • @charles:computer.surgery (me)
• Clients obviously don't need to canonicalize request content, but if they send non-canonicalizable JSON then they get a M_BAD_JSON
  • @tulir:maunium.net

Some additional context:

• Canonicalizing the JSON is unrelated to how you send/receive JSON. You can send/receive it in whatever way is valid JSON, but when you sign/verify, you use the canonical JSON encoding for whatever the JSON values are.
  • @hubert:uhoreg.ca

Solution

IMO this section of the spec needs to be reworded so that it doesn't sound like it's making any requirements on the wire format and that it only applies to some endpoints, as these details are currently causing confusion. Perhaps something like this:

Before signing or verifying JSON data, servers MUST strictly enforce that the JSON data can be canonicalized. For example, data received by servers that cannot be canonicalized can result in 400 M_BAD_JSON for some client endpoints or in events over federation being discarded.

"Before signing or verifying JSON data" clarifies that this only applies to signing/verifying and not data over the wire, and the modified placement of "For example" clarifies that the following text is informative rather than normative, and is expected to only apply in a few places, and allows the rest of the spec to describe those places.

Relatedly, I think it would be very helpful to explicitly state somewhere in the spec that data over the wire is never required to be in canonical format. Currently, this is merely implied by the spec never stating that data over the wire must be canonicalized, which makes it hard to know for sure since it doesn't say the opposite either and there is no section in the spec to point to as a reference. I think this would help mitigate confusion when reading/interpreting other parts of the spec as well.

Start with the canonical JSON sections in the v1.11 appendices and rooms/v11, then review the cited discussion about signing, verification, and wire format. Rework the normative wording so its scope and endpoint behavior are unambiguous, and state whether wire data must be canonical; done means the revised text clearly distinguishes canonicalization from transmission format.