matrix-org / matrix-org/matrix-spec
[MSC stub] Tagging binary data
Nobody has claimed this yet.
- Dominant language
- HTML
- Stars
- 330
- Forks
- 150
- Avg merge
- 2h 21m
- Merged PRs (30d)
- 3
Description
(Dumping my thoughts here of what i discussed with someone else in a room)
#### Proposal
This idea is for a matrix-"canonical" way of tagging binary data in a native-but-compatible form.
Currently in JSON you'd encode some binary data as base64, but in CBOR, you might want to have this be "raw binary" (as is allowed in that spec)
Basically, for JSON then, you can use an object with 1 key inside of it, where the key is prefixed with `_`, and whatever after notes the data encoding.
Example formats could be;
`_b32` (base 32)
`_b64` (base 64)
`_0x` (hexadecimal)
`_0o` (octal)
`_0b` (binary)
`_u` (as-is UTF-8 data, but recognised as binary)
This formatting method is more or less (partially) inspired by Python-esc and Rust-esc prefixing (`123i32`, `0xDEADBEEF`, `u"hi, this is unicode"`)
This would appear as following:
Take the following abstract object;
```json5
{
"key":
}
```
this could then be JSON-encoded into
```json5
{
"key": {"_b64": ""}
}
```
The encoder could select an algorithm that makes most sense, maybe the data is more easily packed as b64, maybe as b32, maybe as hexadecimal, or maybe it just needs to be UTF-8.
CBOR and some other formats might represent this data as "native" binary.
#### Reasoning
The reason for this canonical binary format is to allow a "Binary" type in the spec, this could open it up for a whole array of functionality, be it interests from the foundation, or outside, where binary blobs could be encoded in an easily-formattable way, if this is "canonicalised", deserializers could allow to pick up on this for any array of programming languages that wish to interface with matrix, in a uniform way.
The different encodings allow different "efficient" methods of encoding that data, and the object-wrapping allows the key-"value" abstraction to make deserializing/serializing efficient across a myriad of programming language.
#### Potential problems
On its surface, this does not preserve "roundtrip" information, an deserialized object with binary data might be serialized differently, so if roundtripping information is to be preserved, languages would have to add an extra "tag" to the binary data that tells which encoding it used when read, this is only required when the application, at that moment, has an interest in preserving this information, if the data is only intended to be consumed, it could be "lossily" read.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No implementation files or tests are named. Start by reviewing the proposal and comparing JSON's base64 representation with CBOR's native binary support, then determine whether a canonical Matrix representation is sufficiently specified. Done means the format, encoding rules, and round-trip behavior are agreed and documented in the specification.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- json
- Domain
- backend-api-design
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100