multiformats / multiformats/multicodec

Recursive structures of primitive codecs

Open
#174 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
383
Forks
221
Avg merge
24d 6h
Merged PRs (30d)
1

Description

Instead of simply enumerating every possible codec, why don't we define an exhaustive set of primitives and composing operations which can be assigned an id for reduced size? At present they're hardly self-describing, an implementation has to already know how to decode a codec.

In essence, the current multicodec table would be given an additional column representing decomposition into more primitive types, and codes assigned to more primitive types and compositional operations. This way, an implementation that doesn't know a higher-order codec can fetch an up-to-date table and know how to decompose it. A "primitive" is defined as a row that lacks a decomposition column. Initially, all codecs are primitives, but as decomposition encoding improves, they can be redefined as higher-order without affecting older implementations. Older implementations already have logic defined for decomposing former primitives, while newer implementations either define similar special decoders or defer to the decomposition.

For some vague examples of what this might look like:

name tag code composition description
compact composition ? byte_order:{be, le} bit_order:{be, le} bits:int Bit-for-bit encoding of an integer of arbitrary size
be-uint8 memory ? compact(be, be, 8) Big-endian byte order, big-endian bit order, 8-bit unsigned integer
ones composition ? bits:int offset:int $encodes:int $value:(2**bits - offset) One's complement signed integer
ieee-754-binary32 memory ? sign:bit exponent:ones(be, 8, 127) fraction:compact(be, be, 22) IEEE 32-bit floating point

The most primitive type would be a bit, and compositions would emerge from operations which consume those bits. Since the multicodec table is already filled with some highly compositional types under this scheme, maybe it'd be better to create a new project?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The issue names no implementation files, entry points, or tests. Start by reviewing the current multicodec table and its existing compositional types, then define the primitive and composition model described here. Done would require an agreed design, encoding rules, compatibility approach, and a decision on whether this belongs in multicodec or a new project.

Written by the indexing model from the issue text.

Assessment

Domain
backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.