FilOzone / FilOzone/filecoin-services
FRC: Incrementally verifiable CommP downloads with recovery
- Dominant language
- Solidity
- Stars
- 8
- Forks
- 25
- Avg merge
- 14h 38m
- Merged PRs (30d)
- 12
Description
Coming out of https://github.com/filecoin-project/curio/pull/864 from @magik6k, capturing here so we actually do this when time permits.
1. SP exposes endpoint, e.g. `GET /piece/{pieceCID}?tree={layer}`, returning intermediate CommP tree nodes at a specified granularity (e.g. `leaf32m` = each node covers 32 MiB of data)
2. Client (another SP, SDK, FilBeam, etc.):
a. Verifies the returned layer hashes up to the known PieceCID root (client can now trust SP's layer data)
b. Stream downloads the piece in layer-sized chunks, verifying each chunk's subtree against the known intermediate node
c. On verification failure, discards only that chunk and retries from that offset (potential recovery)
3. Benefits: fail-fast on bad bits, resume-friendly downloads, no wasted bandwidth on corrupted transfers
Existing implementation of this idea: https://github.com/CIDgravity/gw/blob/main/ributil/repair_car.go (multi-layer recovery from bitflips in CAR streams)
- Go implementations exist for tree operations; need JS/TS equivalents
- Curio already builds these trees internally during CommP calculation
- The idea of saving and using intermediate trees has been discussed for some time, including as part of the CommPact proposal so there is code in various places for this
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading the existing multi-layer recovery implementation in ributil/repair_car.go and the Go tree-operation code referenced in the issue, then inspect where Curio builds CommP trees. Define the endpoint and JS/TS verification flow around the stated layer hashes, chunk verification, and offset retries; done means corrupted chunks can be discarded and recovered without re-downloading the whole piece.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, javascript, typescript
- Domain
- api, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100