FilOzone / FilOzone/synapse-sdk

Add streaming download support

Open
#699 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
25
Forks
32
Avg merge
3d 4h
Merged PRs (30d)
22

Description

The SDK currently supports streaming uploads but not streaming downloads. The download path accumulates the full response in memory (via the accumulating reader in synapse-core), which means pieces larger than available process memory can't be downloaded.

## Background

There used to be a \"Retriever\" architecture that returned a \`fetch\` \`Response\` you could stream from, but that was removed and replaced with the current accumulating approach. This needs to be revisited.

## Proposed API

The main complexity is CommP validation during download. Two approaches have been discussed:

1. **Error on last chunk** — the stream errors if CommP validation fails at the end *(preferred)*
2. **Post-stream validation callback** — a promise that resolves after stream completion to indicate whether CommP matched (used in FilBeam)

Consensus is that erroring on the last chunk is more idiomatic with WebAPI Streams, with the ability to **opt out of validation** for cases where it isn't needed.

## Related

A merkle inclusion proof API or header for byte range piece retrievals from Curio is also a TODO (client and server) if we're going to encourage or demonstrating byte range retrievals.

Contributor guide

Open the contributing guide

Research direction

Start by tracing the download path and the accumulating reader in synapse-core, then compare it with the removed Retriever architecture. Decide how WebAPI Streams should report CommP validation, including the proposed opt-out, and verify the behavior for large downloads and the final validation result.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
api
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.