n0-computer / n0-computer/iroh-docs
Optimize document store and sync algorithm
@rklaehn is already working on this.
Since Mar 27, 2024.
- Dominant language
- Rust
- Stars
- 74
- Forks
- 16
- PR merge metrics
- No merged PRs in 30d
Description
The sync algorithm currently reads large parts of the store in to memory. E.g.
// TODO: This is hugely inefficient and needs to be optimized
// For an identity range that includes everything we allocate a vec with all entries of
// the replica here.
let local_values: Vec<_> = self
.store
.get_range(range.clone())?
.collect::<Result<_, _>>()?;
this will be fine for a few 1000 entries, but it won't scale for really large documents. We should make sure that iroh provides few but load-bearing primitives, and currently iroh docs (unlike e.g. blobs) are not load-bearing. As soon as you build something big on top, they will crumble.
I think we can come up with a trait for the store that allows implementing an efficient mem store and disk store, and refactor the sync algo (process_message) to use this trait.
The trait should
- make it possible for the store to cache fingerprints (the current one does this as well)
- provide an interface that allows pivot selection without loading all keys and values into mem
- be sync (no async here, since that will be hell. In the worst case we drive this thing from an actor that has it's own thread)
- support efficient prefix replacement like in the original willow design
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.