feat: CAR-stream session for @helia/block-brokers (one gateway request per root)
- Dominant language
- TypeScript
- Stars
- 1.3k
- Forks
- 159
- Avg merge
- 2h 11m
- Merged PRs (30d)
- 16
Description
## Proposal
The trustless-gateway broker fetches one `?format=raw` request per block. For a many-small-block DAG (a sharded HAMT directory is hundreds of tiny blocks) that is hundreds of round trips. The graph walker's lookahead overlaps them, but the per-request latency still dominates.
A session that issues one `?format=car&dag-scope=all` request per root and serves the walker's per-block `retrieve()` calls from the streamed CAR collapses that to a single request. Blocks stay hash-verified through the session's `validateFn`, and any block the CAR omits or delivers corrupt falls back to a single `?format=raw` fetch.
## Impact
One streaming request instead of N for any CAR-endpoint-backed retrieval: `@helia/verified-fetch` CAR responses and `@helia/car` export over a trustless gateway. In a local bench (249-block sharded DAG, 25ms/request) the single-CAR path ran ~24x faster than per-block, with byte-identical output.
## The design question
`NetworkedStorage.createSession(root, options)` (`@helia/utils` 2.5.2) passes only `options` to `broker.createSession(options)`, so the broker session never receives the root. A CAR-stream session needs it to open the stream. Options, in order of preference:
1. Pass `root` to `broker.createSession(root, options)`. Small signature change, makes root-scoped streaming brokers first-class.
2. Infer the root from the first `retrieve(cid)` (the walker's first get is the root). Works today, but it is implicit coupling.
3. Leave it, and I keep this in my own project.
It would not subclass `AbstractSession` (that is per-block, per-provider, racing); it implements `SessionBlockBroker` directly with a no-op `addPeer`. Want to confirm that shape is acceptable before I open a PR.
Working implementation (as a `BlockSource` in my project, same pump/index/verified-gap-fill logic): https://github.com/SgtPooki/ipfs2foc/blob/main/packages/core/src/car-stream-source.ts . Happy to share the bench too.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with NetworkedStorage.createSession in @helia/utils and the broker.createSession interface, then read AbstractSession and SessionBlockBroker to understand the existing session boundary. Compare the three root-passing options and the linked car-stream-source.ts implementation; done requires an agreed API shape for root-scoped CAR streaming before implementation begins.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- distributed-systems, networking
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100