libp2p / libp2p/py-libp2p

QUIC-Based P2P File Sharing with DHT vs Non-DHT Peer Discovery

Open
#1,277 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
624
Forks
256
Avg merge
1d 34m
Merged PRs (30d)
47

Description

### Summary
Build a practical file-sharing MVP on `py-libp2p` for a small peer set (target: 5 peers, file sizes up to 5 GB), and evaluate two peer discovery strategies:

1. Kademlia DHT-based discovery (baseline)
2. Non-DHT discovery (fast path, app-level)

This issue defines the scope, architecture, and measurable outcomes for comparing both approaches.

#### Why This Issue
- We are trying to build a Lean Pylibp2p for developer, ( developer friendy) , we will not be giving a lot of features, but features that they need in default , which make it easy to use.
- for testing this we will make MVP that works for real file transfer workloads (multi-GB files).
- We need to quantify discovery latency and reliability, not assume one method is better.
- We need a baseline (DHT) and a controlled alternative (non-DHT) for design decisions.

#### Goals
- Implement an end-to-end file sharing flow using `py-libp2p` + Bitswap over QUIC.
- Support direct peer-to-peer download using share metadata (ticket/direct connect).
- Implement and compare:
- DHT discovery path: `file_hash -> provider peer(s)` via Kademlia.
- Non-DHT discovery path: application-managed provider lookup.
- Produce empirical measurements for both approaches.

#### Scope
#### 1. Transfer Path
- Use QUIC transport for all peers.
- Chunking and CID verification through current Bitswap/DAG flow.
- Resume-friendly download behavior for large files.
- Chunking strategy experiments:
- Parallel Chunk Fetch
- Disk Streaming Reassembly

#### 2. Discovery Path A: Kademlia DHT
- Publish provider records for shared files.
- Query providers from DHT before dialing.
- Measure cold and warm lookup behavior.

#### 3. Discovery Path B: Non-DHT
- Implement app-level provider discovery with one of:
- Share ticket direct-connect path (`peer_id + addrs + file_root_cid`)
- Lightweight provider index service for `file_hash -> providers`
- Query this path first for fast lookup.

#### 4. Comparison Framework
- Same topology and workload for both paths.
- Repeat test runs and report p50/p95 values.

#### Expected Outcomes
- A measurable baseline for DHT discovery in small-peer file sharing.
- A measurable non-DHT alternative for low-latency provider resolution.
- Clear recommendation for default discovery path in MVP and fallback strategy.

#### Note
- QUIC-only transport is intentional for this MVP to reduce handshake complexity and transport-level HOL issues.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the existing QUIC transport, Bitswap/DAG transfer flow, and Kademlia discovery support in py-libp2p. Define the MVP test topology and workload, then compare the DHT and application-managed provider paths using repeated runs; done means p50/p95 measurements, reliability results, and a documented default and fallback recommendation.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems, networking
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.