Manifest Format

Many features are gated on the basic design of the Intermodal manifest. So let's get started on it right away.

## Desiderata

- Allow integrity checking. Given a manifest, an accompanying release can be checked for integrity using the manifest. This will require including secure hashes of accompanying files in the manifest.

- Hashing the manifest should give a secure hash that uniquely identifies the contents of the release.

- Multi-level manifest. A lower-level manifest should commit to the contents of a release. A higher -level manifest should commit to both the lower level manifest, as well as any files containing signatures over the lower level manifest. It would be nice to only have one manifest, but since data can't self-sign or contain hashes to itself, it seems necessary to have at least a two-level manifest so that we can produce a hash that uniquely identifies a collection of files, as well as signatures and other commitments to that collection of files.

I'm thinking about calling the lower-level manifest the "content manifest" and the lower-level "bundle manifest". I'm definitely open to naming suggestions though. Other ideas are "file manifest" and "root manifest".

## Why not use BitTorrent metainfo?

- BitTorrent v1 uses SHA1, which is insecure.

- BitTorrent v2 uses a custom tree hash that is vulnerable to attack if the length of the content is not included.

- Bencoding is not a particularly popular encoding format.

## Why not use the web packaging format?

The web bundle format is a single-file format, so it would be impossible to use natively with BitTorrent, which is an important transport.

## Out of Scope

To keep things simple, it would be a good idea to limit the scope of the initial manifest design as much as possible. Things that we should consider for the design, but not worry about the details:

- Metadata. Structured metadata can be included in a file that the content manifest commits to.

- Signatures, timestamps, and related functionality. A two-level manifest leaves open the ability to include files that are signatures over the hash of the content manifest, which are committed to by the bundle manifest.

## In scope

- Manifest format. I'm thinking either CBOR or messagepack. They are both lightweight, binary, schemaless formats with an object model that is similar to JSON. The web packaging format uses CBOR, so that's what I'm leaning towards. Keybase's [saltpack](https://saltpack.org/), however, uses messagepack, so that's a contender too.

- The hash function. Since manifests will have to include secure hashes, we should pick a hash function. I'm leaning towards [BLAKE3](https://github.com/BLAKE3-team/BLAKE3), although someone could probably talk me out of it. BLAKE3 is extremely fast, supports random access and streaming verification, and has a strong rust implementation, all of which are nice features. On the downside, it is very new, and uses a reduced strength construction. However, that reduced strength is argued to [not be vulnerable now or in the future](https://eprint.iacr.org/2019/1492.pdf).

- Whether the manifest should be flat or a tree. Nested will be more compact when there are long directory names with many entires, but is more complex. Nested doesn't explicitly encode path separators, which I think is a bonus.

```
flat: {"foo/bar": "BAR_HASH", "foo/baz": "BAZ_HASH"}
tree: {foo: {bar: "BAR_HASH", baz: "BAZ_HASH"}}
```

BitTorrent V2 uses a tree, so that's what I'm leaning towards.

- Where to put the manifest in a release. Since it seems likely that we'll eventually want multiple files, I'm thinking that putting everything into a subdirectory is a good idea, either `imdl/` or `intermodal/`.

For example, if we go with CBOR, the structure could be:

```
intermodal/root.cbor # root manifest
intermodal/content.cbor # content manifest
intermodal/signatures # signatures over hash of content.manifest
intermodal/timestamp.ots # open timestamps timestamp
intermodal/metadata.cbor # metadata
intermodal/README.txt # human readable info about intermodal
```

## Postscript

A very weird but nonetheless interesting choice of format would be [FIDL](https://fuchsia.dev/fuchsia-src/development/languages/fidl), Fuchsia's IPC system:

- Modular compiler and language bindings.
- Message encoding is canonical — there is exactly one encoding for a given message.
- Supports both fixed-size messages where space is important, and extensible tables and unions where schema evolution is important.
- Defined to be little endian and uses natural alignment, so no portability issues.
- Zero copy and zero parse, by storing variable sized members out-of-line. Can be memory mapped with LMDB or mmap(2) for extremely fast access.
- Certain types can be introspected without access to a schema, namely tables and unions.
- Could make it fully introspectable where needed by including hash of schema as first element of messages.

Another less wacky choice would be [flatbuffers](https://google.github.io/flatbuffers/). Flatbuffers also support zero copy and zero parse deserialization.

## Misc

- Would like to compare the encoding choices of flatbuffer and FIDL.
- Would like to add some kind of `link` attribute, that would indirect through a hash.
- Unsure of how to approach "canonical encoding". Forcing the format to output a buffer in a canonical format is inflexible. A more flexible approach would be to calculate hashes over logical traversals of the physical data, so the physical data may be in multiple forms, but hash to the same value, as long as the traversal doesn't change.
- [Flatbuffers binary format documentation](https://github.com/dvidelabs/flatcc/blob/master/doc/binary-format.md)
- [fleetfs flatbuffers schema](https://github.com/fleetfs/fleetfs/blob/master/flatbuffers/messages.fbs)
- [proc macro attribute parser](https://github.com/TedDriggs/darling)
- [syn helper](https://github.com/mystor/synstructure)

No implementation file, test, or entry point is identified. Start by reviewing the manifest requirements and the proposed CBOR, MessagePack, BLAKE3, flat/tree, and release-layout options; the design is done when these choices are resolved into a defined manifest specification and implementation plan.

Written by the indexing model from the issue text.