API Design Discussion

As per https://github.com/ipfs/notes/issues/436 I would like to use ipfs-lite as a foundation for providing IPLD storage & replication system. At the same time I would like to use this as an opportunity to explore / design API fit best for that system instead of trying to provide backwards compatibility with [`ipfs.dag`][] and [`ipfs.block`][] APIs that were designed for different set of constraints and use cases. Here is a my short (constraint / wish) list:

- [ ] Interops with (new) IPLD stack instead of mimicing or wrapping it.

> E.g [`ipfs.dag`][] under the hood creates IPLD BLocks, serializes them and stores them. Instead we should just take serializable IPLD [`Block`][] and store it's bytes.

- [ ] Interop layer should be extremely minimal to reduce coordination to a bare minimum.

> E.g. Block API could change and may have many bells and whistles, instead of assuming `Block` instances we should have a minimal `Block` interface that can be serialized, identified, ...

- [ ] Interop layer should enable case specific `Block` implementations
> This should be achieved from the above goal, but I wanted to calling it out explicitly.

- [ ] Designed for multi-threaded use.

> Copying things across threads in JS can be costly. API should consider that and create a room for optimizations. That means instead of blocks telling how to consume data they should allow flexibility for a consumer to choose a best way to consume it. E.g Web [`Response`](https://developer.mozilla.org/en-US/docs/Web/API/Response) is good example allowing consumer to consume it as stream, blob, text etc... while `AsyncIterable` is not because:
> - It can be read only in one way
> - It is impossible to tell if underlying content is in memory or being allocated as it's being consumed.
> - It provides no information about the amount of content.
> - It provides no context regarding if individual chunks can be consumed ([transferred][Transferable]) or need to be copied.
> - Isn't peekable

- [ ] First class batching

> - js-ipfs uses `AsyncIterables` to put consumer in control providing (pull interface).
> - New [car](https://www.npmjs.com/package/datastore-car) format is used to pack multiple blocks and push them out in one go (push interface).
>
> What we need is a first class primitive for representing multiple blocks which
> - Allows consumer to **pull** on it's own schedule.
> - Allow producer to **push** multiple blocks on it's own schedule.
> - Allows application author decide how tightly two should coordinate.
>
> This may [seem familiar](https://en.wikipedia.org/wiki/Communicating_sequential_processes)

[`ipfs.dag`]:https://github.com/ipfs/js-ipfs/blob/master/docs/core-api/DAG.md
[`ipfs.block`]:https://github.com/ipfs/js-ipfs/blob/master/docs/core-api/BLOCK.md
[`Block`]:https://github.com/ipld/js-block
[Transferable]:https://developer.mozilla.org/en-US/docs/Web/API/Transferable