More efficient blob announcement using bloom filters
Nobody has claimed this yet.
Assessment
- Difficulty
- 5/5
- Estimated time
- Over a week
- Newbie friendliness
- 20/100
- Issue type
- Feature
- Clarity
- Needs clarification
- Activity status
- Stale
- Tech stack
- python
- Domain
- distributed-systems, networking
Research direction
The issue describes a protocol change for blob announcements and findNode responses but names no files or tests. Start by locating the DHT implementations of iterative find-and-store and findNode, then clarify the bloom-filter storage and lookup design. Done means bulk announcements and fallback lookups work correctly with coverage for the new behavior.
Written by the indexing model from the issue text.
Description
tldr
When announcing a lot of blobs, you'll often be announcing many blobs to the same peer. Instead of doing many iterative-find-and-store cycles (one for each hash), do a single cycle and store a bloom filter of all the hashes together.
When answering a findNode call, if k peers aren't found by checking exact hash matches, the node will check it's bloom filters for matches.
details
bloom filter idea:
split all hashes in k shards
find k peers for each shard
announce a bloom filter for every 200 items for k peers in each set
split all hashes i
to announce many hashes:
- determine k closest peers for all hashes, build set of hashes per peer
- build bloom filter for hashes to announce to each peer
- send filters for each peer
to find a peer for a hash:
- find the k closest to the hash in the rt
- send findvalue requests to them
a. peer checks datastore for the hash, returns k peers if hit
b. peer checks stored bloom filters for if they contain the hash, returns up to k peers that have hits
- Dominant language
- Python
- Stars
- 7.2k
- Forks
- 493
- PR merge metrics
- No merged PRs in 30d
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from lbryio/lbry-sdk
-
Difficulty 5/5 Over a week Newbie friendliness 15/100
-
Difficulty 4/5 3-5 days Newbie friendliness 20/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 35/100
-
Difficulty 5/5 Over a week Newbie friendliness 20/100
-
Difficulty 4/5 3-5 days Newbie friendliness 25/100
Similar issues
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
bancolombia/sentinel#23 ·
-
test md OpenCI
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
-
integration:quickjs org:external priority:backlog topic:code-interpreter topic:middleware type:feature
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
langchain-ai/deepagents#6450 ·
-
bug client
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 74/100