Design: Local vs Remote Storage, Whither The BlockService

This issue is designed to provoke discussion around design issues related local vs remote storag as it relates to efforts to write new interfaces for working with DAG structures in go-ipld-prime. It's mostly a documentation for context for the team working on this, but I think laying out these concepts may help others in the future in thinking about designing abstractions for IPFS.

# Overview

In a sense, the entire IPFS network can be thought of as a single, distributed file storage system. In fact, this is the ideal goal we are always striving for. We should be able to imagine a single user of IPFS having access to the entirety of what is stored on the network as if it were local.

In practice, keeping a local copy of some data makes sense both for fast access and because the user may want take responsibility for providing data to the network. In the IPFS software, the local copy is the blockstore.

We might think of the blockstore as similar to the web browser cache. Just like a user doesn't specify if a web page is cached locally or must be fetched remotely when they type in a URL, at the command like or HTTP api level, from a user perspective, we should strive to never worry about where data is coming from.

At the same time, unlike a web browser cache, when we receive data or add it from the command line, we may also become a host for that data on the network. Writing to the local blockstore is therefore only one part of a write operation. The second part is to provide the written cids to the DHT.

So to illustrate the symmetry of local / remote on read vs write:

Operation | Local Component | Remote Component
---------- | ------------------ | ---------------------
Read | Load from blockstore | Read from network
Write | Write to blockstore | Provide To Network

Importantly, all new interfaces that work with DAGs, for reading or writing, must insure they perform both parts of a read and write (unless specifically told not to)

# Level Of Abstraction, Designing New Services

Currently the key service that performs the local/remote abstraction is the BlockService. The block service abstracts at the level of Block read/writes. When I add a DAG, I call AddBlocks / AddBlock on the block service for all blocks in the DAG, which writes each block individual then calls out to Bitswap to provide each block individually. When I fetch a DAG, I call GetBlock / GetBlocks on the block service for all blocks in the DAG, which attempts to load the block locally then calls out to Bitswap to fetch the blocks from the network (which ultimately saves the blocks to the local store and provides them as well).

So we have:
Blockstore -> Local Block Read/Write
Bitswap -> Remote Block Read/Write
BlockService -> Local/Remote Block Read/Write Abstraction

The system is coherent and works as long as local and remote fetching or providing works at the block level. However, consider the things this locks us into if we care about DAGs:
- How do we provide only the root of a DAG? (often something we want to do to make the DHT less crowded)
- How would this work if we're fetching with Graphsync or a combination of graphsync and bitswap?

We've actually tried to move the providing to a different place and offer DAG roots only (see github.com/ipfs/go-ipfs-provider) but this is still experimental. And the tight integration of blockstore / bitswap / blockservice has made it hard to deliver on projects like this.

As we design our interfaces, I don't think we should try to change any of this immediately, but we should design them with an eye towards moving abstractions up in the future.

# Hard questions in interface design

Currently we have [go-fetcher](github.com/ipfs/go-fetcher) and [go-dagservice](github.com/ipfs/go-dagservice) two in progress repos for working with IPLD prime data. Reads however often correspond with writes and i'm not 100% sure about keeping these seperate.

Here are some questions I am thinking about as we look at designing these two services:
- How would I add a whole dag and specify provide only a root?
- Assuming eventually IPLD prime finishes implementation of FocusTransform and WalkTransforming, how should I be able to access these interfaces?
- Important question: should I be able to perform a transform (which potentially digs deep into the DAG) on a DAG I haven't fetched entirely locally? I think maybe I should be able to do this.
- are reads and writes seperate enough that we can make them seperate interfaces?