Qri needs a long-term plan for operating on IPFS, libp2p protocols out-of-process
- Dominant language
- Go
- Stars
- 17.1k
- Forks
- 3.2k
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 11
Description
Most of our users come from the world of data science, and don't (yet) have IPFS installed independent of our binary. Then @momack2 gave Qri Desktop a spin, and it turns out we don't play very nice with other IPFS's 😞:
https://github.com/qri-io/qri/issues/1296
https://github.com/qri-io/desktop/issues/535
Right now Qri wants the repo lock so it can run go-ipfs in-process. Long term, we really shouldn't need to do that, and it should instead be possible to run IPFS somewhere else. Needing to choose between Desktop applications isn't ok.
That said, this problem is not small. There are two immediate blockers to us operating on IPFS out-of-process:
1. our dsync algorithm needs to know if blocks are local or not. I've filed #6726 for this one, but have no idea how this would work over a Unix Socket/HTTP API.
2. we register at least 3 custom libp2p protocols, and tune the connection manager to prioritize connections to qri peers.
On the upside, we do have cursory support for operating on IPFS over the HTTP API, and use this exact setup in our cloud backend services. But with the two above issues we've had to carefully carve our backend architecture to support these deficits by having some services that operate on in-process repos, and some that operate over HTTP with a lot of configuration and reduced functionality.
Finally performance is an obvious concern here. Much of Qri's performance depends on fast access to the local blockstore (SQL JOINs will access blocks _once per row_ at the moment).
I'd love to get a discussion going to figure out the right course of action is so we can start to work in a good roadmap. I'm not sure what the right end state for a project like ours even _is_, given that there's no way to register a libp2p protocol without access to the host itself, and by extension, the repo lock (unless I'm missing something crucial. I'd love to be missing something crucial).
Contributor guide
Assessment
This issue has not been assessed yet.