FilOzone / FilOzone/ipfs2foc

[Feature]: parallel p2p retrieval for console prepare (verified-fetch)

Open
#59 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
1
Forks
0
Avg merge
5d 21h
Merged PRs (30d)
1

Description

**Description**

Console prepare pulls every root as one streaming `?format=car` request from the single configured gateway, and the per-block gap-fill fallback hits the same host (`app/src/commp.ts`, `packages/core/src/car-stream-source.ts`). Retrieval concurrency is pinned to the hash pool: `CONCURRENCY = HASH_POOL_SIZE = 4` in `app/src/app.tsx`. #57 made a 2,159,085-CID run manageable in the UI, but a live run against that inventory sat at ~400 ready after a few minutes — roughly 2 CIDs/s, which extrapolates to nearly two weeks for the full list, with every byte routed through one central endpoint.

**Impact**

Retrieval throughput is now the binding constraint on million-CID runs, and one gateway is both the bottleneck and a single point of rate limiting for the whole migration.

**Next steps**

- Fetch blocks in the browser with `@helia/verified-fetch` over js-libp2p so retrieval spreads across providers instead of one host; verification stays cryptographic per block. #19 evaluated the same move for the CLI.
- Decouple retrieval concurrency from the hash pool: hashing is CPU-bound (4 workers is right), but many more fetches can be in flight feeding those workers, especially with a 63 KiB median root.
- Invariant to hold: the `sourceUrl` handed to providers, and the canonical CAR bytes it serves, must not change — how the console verifies locally is independent of what the provider pulls (`docs/onchain.md`).

Contributor guide

Open the contributing guide

Research direction

Start by tracing console prepare through app/src/app.tsx, app/src/commp.ts, and packages/core/src/car-stream-source.ts, then read docs/onchain.md for the sourceUrl and CAR-byte invariant. Compare the proposed browser retrieval path with @helia/verified-fetch and the concurrency assumptions. Done means provider retrieval is distributed, hashing and retrieval concurrency are independent, and the stated invariant remains intact.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
frontend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.