n0-computer / n0-computer/iroh-docs

Optimize document store and sync algorithm

Open
#59 1 comment 0 reactions 1 assignee View on GitHub

@rklaehn is already working on this.

Since Mar 27, 2024.

Dominant language
Rust
Stars
74
Forks
16
PR merge metrics
No merged PRs in 30d

Description

The sync algorithm currently reads large parts of the store in to memory. E.g.

// TODO: This is hugely inefficient and needs to be optimized
// For an identity range that includes everything we allocate a vec with all entries of
// the replica here.
let local_values: Vec<_> = self
    .store
    .get_range(range.clone())?
    .collect::<Result<_, _>>()?;

this will be fine for a few 1000 entries, but it won't scale for really large documents. We should make sure that iroh provides few but load-bearing primitives, and currently iroh docs (unlike e.g. blobs) are not load-bearing. As soon as you build something big on top, they will crumble.

I think we can come up with a trait for the store that allows implementing an efficient mem store and disk store, and refactor the sync algo (process_message) to use this trait.

The trait should

  • make it possible for the store to cache fingerprints (the current one does this as well)
  • provide an interface that allows pivot selection without loading all keys and values into mem
  • be sync (no async here, since that will be hell. In the worst case we drive this thing from an actor that has it's own thread)
  • support efficient prefix replacement like in the original willow design

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.