Next Steps

Since #1 is clogged with all the many comments I open a new issue here. Feel free to continue the discussion below and I'll keep the following updated as things develop. Also feel free to create separate issues / repos to coordinate and I'll add the relevant links below.

Next steps (networking, stalled – please see the “storage” section below):
* [ ] We could use some documentation for the `py-libp2p` library
* See issue https://github.com/zixuanzh/py-libp2p/issues/35 for status
* Current work: https://github.com/libp2p/py-libp2p/pull/330
* [ ] Documentation for the Bitswap protocol would be very useful as well – and not just for py-ipfs
* See issue https://github.com/ipfs/js-ipfs-bitswap/issues/21 for status
* Currently available information:
* Protocol: https://github.com/ipfs/specs/blob/master/BITSWAP.md
* API: https://ipfs.github.io/js-ipfs-bitswap/#bitswapget
* [ ] Finally, some documentation on the currently used DHT would be nice
* See https://github.com/ipfs/camp/blob/master/DEEP_DIVES/02-scaling-up-the-dht.md for some relevant links and current developments
* [ ] Implement the Bitswap protocol in Python 3
* Current `py-ipfs-bitswap` library: https://github.com/AliabbasMerchant/py-ipfs-bitswap
* Current status/discussion: https://github.com/AliabbasMerchant/py-ipfs-bitswap/issues/1
* → Having this would allow us to implement the `ipfs block *` API for fetching blocks of nodes we are connected to – fetching blocks of non-connected nodes needs the DHT.
* To interact with go-IPFS you can start it with `ipfs daemon --disable-transport-encryption`, but note that you will not be able to connect to any *regular* peers until one of the transport encryption methods is implemented
* [X] (*Low priority*) Improve the [`multistream-select` code of `py-libp2p`](https://github.com/libp2p/py-libp2p/tree/master/libp2p/protocol_muxer) to support actually dialing other nodes ([**MOSTLY FIXED UPSTREAM**](https://github.com/libp2p/py-libp2p/commits/master/libp2p/protocol_muxer/multiselect_communicator.py) – [`ls` is still missing](https://github.com/libp2p/py-libp2p/blob/30aeb35122883e51f6aaa41b8ba704e44e4ae8a2/libp2p/protocol_muxer/multiselect.py#L51) and an `mss-nc` implementation could still be useful)
* Write a [`mss-nc`](https://github.com/whyrusleeping/mss-nc) like utility on top of this code to demonstrate that you are able to connect to `go-ipfs` nodes and negotiate
* Here's some very simple sample code demonstrating the main mode of MSS:
```py
import socket
s = socket.socket(socket.AF_INET)
s.connect(("127.0.0.1", 4001)) # The connect will already exist
s.sendall(b'\x13/multistream/1.0.0\n') # Send your supported version of MSS
s.recv(1024) # → b'\x13/multistream/1.0.0\n' – Receive supported version of MSS by other party & validate!
s.sendall(b"\x0d/secio/1.0.0\n") # Request the protocol you'd like to upgrade too
s.recv(1024) # → b'\0x0d/secio/1.0.0\n…' – Confirmation that protocol is available + Protocol data OR
# → b'\0x03na\n' – Protocol was Not Available
```
* The binary values at the start are varints and you need to read them byte-by-byte until you're done decoding them, then read the remainder of each message lines based on the received length value
To do this you'll need to create an async stream based version of https://github.com/fmoo/python-varint/blob/master/varint.py
* Additionally there is also a special `ls` mode in which MSS will return a list of supported protocols, see https://github.com/multiformats/multistream-select/blob/master/README.md for the complete spec
* Subtask: Figure out how to actually dial a node using py-libp2p and document this using an example program.
* [x] Implement [SecIO](https://github.com/libp2p/specs/tree/master/secio) [libp2p transport security](https://github.com/libp2p/py-libp2p/tree/master/libp2p/security) (easier than TLS, but will be phased out eventually)
* You'll need to coordinate with the py-libp2p guys on transport security modules are added exactly
* Some background with crypto is highly recommended!
* [ ] Implement DHT peer lookups in `libp2p`
* Currently available documentation: https://github.com/libp2p/specs/tree/8b89dc2521b48bf6edab7c93e8129156a7f5f02c/kad-dht
* Go Implementation: https://github.com/libp2p/go-libp2p-kad-dht
* JavaScript Implementation: https://github.com/libp2p/js-libp2p-kad-dht
* For the current status see: https://github.com/libp2p/py-libp2p/issues/150 (also see the referenced PRs on that issue)
* Interesting PRs: [#129](https://github.com/libp2p/py-libp2p/pull/129), [#153](https://github.com/libp2p/py-libp2p/pull/153), [#157](https://github.com/libp2p/py-libp2p/pull/157)
* Contacts: @alexh @zaibon @zixuanzh
* [ ] (*Stretch goal*) Factor out https://github.com/zixuanzh/py-libp2p/tree/master/protocol_muxer into a separate `py-multistream-select` library and update `py-libp2p` to use it (**Easy!**, *stalled* – needs your help!)
* Current `py-multistream-select` library: https://github.com/dheatovwil/py-multistream-select
* Current status: https://github.com/zixuanzh/py-libp2p/pull/101
* [ ] (*Stretch goal*) Implement [TLSv1.3](https://github.com/libp2p/specs/blob/master/tls/tls.md) [libp2p transport security](https://github.com/libp2p/py-libp2p/tree/master/libp2p/security)
* You'll need to coordinate with the py-libp2p guys on transport security modules are added exactly
* Some background with crypto/X.509/TLS is highly recommended!

Next steps (storage, simpler):
1. [x] Port https://github.com/ipfs/py-datastore to Python 3
(Suggestion: Use Python's `lib2to3` and just drop Python 2 entirely.)
* ~~Current port: https://github.com/dheatovwil/datastore~~
2. [x] Convert datastore to use async/await using some library ~~(maybe https://pypi.org/project/aiofiles/ ?) for file access~~ The [`trio`](https://trio.readthedocs.io/) framework is used for async I/O now
3. [ ] Implement a https://github.com/ipfs/go-ds-flatfs compatible backend for the above library
4. [ ] Write a minimal `py-ipfs` “implementation” that can fetch blocks from the local `$IPFS_PATH` directory and expose them with an API similar to what https://github.com/ipfs/py-ipfs-http-client currently offers (goal here is to eventually have a drop-in replacement)
* In progress by @alexander255 (no public code yet, most work happens in py-datastore)
5. [ ] Implement a simple Python HTTP server that emulates the [`block/{get,put,rm,stat}` API](https://ipfs.io/ipns/docs.ipfs.io/reference/api/http/#api-v0-block-get) that serves blocks from the local `$IPFS_PATH` directory
* Recommendation: Use the [`trio-quart` ASGI web microframework](https://gitlab.com/pgjones/quart-trio/) for this. (Whatever you choose it will have to be compatible with [trio](https://trio.readthedocs.io/) as that is the AIO framework used in the stack.)
6. [ ] (*Stretch goal*) Implement a badgerds compatible backend for py-datastore
* There is an issue requesting Python bindings for the Go library, but no work has been done yet:
[dgraph-io/badger#984](https://github.com/dgraph-io/badger/issues/984)
6. [ ] *Beyond*: Start integrating [IPLD](https://github.com/bigchaindb/py-ipld/) to expose the UnixFS files stored in those raw blocks…