multiformats / multiformats/multicodec
Recursive structures of primitive codecs
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 383
- Forks
- 221
- Avg merge
- 24d 6h
- Merged PRs (30d)
- 1
Description
Instead of simply enumerating every possible codec, why don't we define an exhaustive set of primitives and composing operations which can be assigned an id for reduced size? At present they're hardly self-describing, an implementation has to already know how to decode a codec.
In essence, the current multicodec table would be given an additional column representing decomposition into more primitive types, and codes assigned to more primitive types and compositional operations. This way, an implementation that doesn't know a higher-order codec can fetch an up-to-date table and know how to decompose it. A "primitive" is defined as a row that lacks a decomposition column. Initially, all codecs are primitives, but as decomposition encoding improves, they can be redefined as higher-order without affecting older implementations. Older implementations already have logic defined for decomposing former primitives, while newer implementations either define similar special decoders or defer to the decomposition.
For some vague examples of what this might look like:
| name | tag | code | composition | description |
|---|---|---|---|---|
| compact | composition | ? | byte_order:{be, le} bit_order:{be, le} bits:int | Bit-for-bit encoding of an integer of arbitrary size |
| be-uint8 | memory | ? | compact(be, be, 8) | Big-endian byte order, big-endian bit order, 8-bit unsigned integer |
| ones | composition | ? | bits:int offset:int $encodes:int $value:(2**bits - offset) | One's complement signed integer |
| ieee-754-binary32 | memory | ? | sign:bit exponent:ones(be, 8, 127) fraction:compact(be, be, 22) | IEEE 32-bit floating point |
The most primitive type would be a bit, and compositions would emerge from operations which consume those bits. Since the multicodec table is already filled with some highly compositional types under this scheme, maybe it'd be better to create a new project?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The issue names no implementation files, entry points, or tests. Start by reviewing the current multicodec table and its existing compositional types, then define the primitive and composition model described here. Done would require an agreed design, encoding rules, compatibility approach, and a decision on whether this belongs in multicodec or a new project.
Written by the indexing model from the issue text.
Assessment
- Domain
- backend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100