ipfs / ipfs/helia

feat: CAR-stream session for @helia/block-brokers (one gateway request per root)

Open
#1,051 0 comments 2 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
1.3k
Forks
159
Avg merge
2h 11m
Merged PRs (30d)
16

Description

## Proposal

The trustless-gateway broker fetches one `?format=raw` request per block. For a many-small-block DAG (a sharded HAMT directory is hundreds of tiny blocks) that is hundreds of round trips. The graph walker's lookahead overlaps them, but the per-request latency still dominates.

A session that issues one `?format=car&dag-scope=all` request per root and serves the walker's per-block `retrieve()` calls from the streamed CAR collapses that to a single request. Blocks stay hash-verified through the session's `validateFn`, and any block the CAR omits or delivers corrupt falls back to a single `?format=raw` fetch.

## Impact

One streaming request instead of N for any CAR-endpoint-backed retrieval: `@helia/verified-fetch` CAR responses and `@helia/car` export over a trustless gateway. In a local bench (249-block sharded DAG, 25ms/request) the single-CAR path ran ~24x faster than per-block, with byte-identical output.

## The design question

`NetworkedStorage.createSession(root, options)` (`@helia/utils` 2.5.2) passes only `options` to `broker.createSession(options)`, so the broker session never receives the root. A CAR-stream session needs it to open the stream. Options, in order of preference:

1. Pass `root` to `broker.createSession(root, options)`. Small signature change, makes root-scoped streaming brokers first-class.
2. Infer the root from the first `retrieve(cid)` (the walker's first get is the root). Works today, but it is implicit coupling.
3. Leave it, and I keep this in my own project.

It would not subclass `AbstractSession` (that is per-block, per-provider, racing); it implements `SessionBlockBroker` directly with a no-op `addPeer`. Want to confirm that shape is acceptable before I open a PR.

Working implementation (as a `BlockSource` in my project, same pump/index/verified-gap-fill logic): https://github.com/SgtPooki/ipfs2foc/blob/main/packages/core/src/car-stream-source.ts . Happy to share the bench too.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with NetworkedStorage.createSession in @helia/utils and the broker.createSession interface, then read AbstractSession and SessionBlockBroker to understand the existing session boundary. Compare the three root-passing options and the linked car-stream-source.ts implementation; done requires an agreed API shape for root-scoped CAR streaming before implementation begins.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
distributed-systems, networking
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.