FilOzone / FilOzone/filecoin-services

FRC: Incrementally verifiable CommP downloads with recovery

Open
#387 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Solidity
Stars
8
Forks
25
Avg merge
14h 38m
Merged PRs (30d)
12

Description

Coming out of https://github.com/filecoin-project/curio/pull/864 from @magik6k, capturing here so we actually do this when time permits.

1. SP exposes endpoint, e.g. `GET /piece/{pieceCID}?tree={layer}`, returning intermediate CommP tree nodes at a specified granularity (e.g. `leaf32m` = each node covers 32 MiB of data)
2. Client (another SP, SDK, FilBeam, etc.):
a. Verifies the returned layer hashes up to the known PieceCID root (client can now trust SP's layer data)
b. Stream downloads the piece in layer-sized chunks, verifying each chunk's subtree against the known intermediate node
c. On verification failure, discards only that chunk and retries from that offset (potential recovery)
3. Benefits: fail-fast on bad bits, resume-friendly downloads, no wasted bandwidth on corrupted transfers

Existing implementation of this idea: https://github.com/CIDgravity/gw/blob/main/ributil/repair_car.go (multi-layer recovery from bitflips in CAR streams)

- Go implementations exist for tree operations; need JS/TS equivalents
- Curio already builds these trees internally during CommP calculation
- The idea of saving and using intermediate trees has been discussed for some time, including as part of the CommPact proposal so there is code in various places for this

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading the existing multi-layer recovery implementation in ributil/repair_car.go and the Go tree-operation code referenced in the issue, then inspect where Curio builds CommP trees. Define the endpoint and JS/TS verification flow around the stated layer hashes, chunk verification, and offset retries; done means corrupted chunks can be discarded and recovered without re-downloading the whole piece.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, javascript, typescript
Domain
api, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.