[Feature]: parallel p2p retrieval for console prepare (verified-fetch)
- Dominant language
- TypeScript
- Stars
- 1
- Forks
- 0
- Avg merge
- 5d 21h
- Merged PRs (30d)
- 1
Description
**Description**
Console prepare pulls every root as one streaming `?format=car` request from the single configured gateway, and the per-block gap-fill fallback hits the same host (`app/src/commp.ts`, `packages/core/src/car-stream-source.ts`). Retrieval concurrency is pinned to the hash pool: `CONCURRENCY = HASH_POOL_SIZE = 4` in `app/src/app.tsx`. #57 made a 2,159,085-CID run manageable in the UI, but a live run against that inventory sat at ~400 ready after a few minutes — roughly 2 CIDs/s, which extrapolates to nearly two weeks for the full list, with every byte routed through one central endpoint.
**Impact**
Retrieval throughput is now the binding constraint on million-CID runs, and one gateway is both the bottleneck and a single point of rate limiting for the whole migration.
**Next steps**
- Fetch blocks in the browser with `@helia/verified-fetch` over js-libp2p so retrieval spreads across providers instead of one host; verification stays cryptographic per block. #19 evaluated the same move for the CLI.
- Decouple retrieval concurrency from the hash pool: hashing is CPU-bound (4 workers is right), but many more fetches can be in flight feeding those workers, especially with a 63 KiB median root.
- Invariant to hold: the `sourceUrl` handed to providers, and the canonical CAR bytes it serves, must not change — how the console verifies locally is independent of what the provider pulls (`docs/onchain.md`).
Contributor guide
Research direction
Start by tracing console prepare through app/src/app.tsx, app/src/commp.ts, and packages/core/src/car-stream-source.ts, then read docs/onchain.md for the sourceUrl and CAR-byte invariant. Compare the proposed browser retrieval path with @helia/verified-fetch and the concurrency assumptions. Done means provider retrieval is distributed, hashing and retrieval concurrency are independent, and the stated invariant remains intact.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- frontend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100