FilOzone / FilOzone/synapse-sdk
Add streaming download support
- Dominant language
- TypeScript
- Stars
- 25
- Forks
- 32
- Avg merge
- 3d 4h
- Merged PRs (30d)
- 22
Description
The SDK currently supports streaming uploads but not streaming downloads. The download path accumulates the full response in memory (via the accumulating reader in synapse-core), which means pieces larger than available process memory can't be downloaded.
## Background
There used to be a \"Retriever\" architecture that returned a \`fetch\` \`Response\` you could stream from, but that was removed and replaced with the current accumulating approach. This needs to be revisited.
## Proposed API
The main complexity is CommP validation during download. Two approaches have been discussed:
1. **Error on last chunk** — the stream errors if CommP validation fails at the end *(preferred)*
2. **Post-stream validation callback** — a promise that resolves after stream completion to indicate whether CommP matched (used in FilBeam)
Consensus is that erroring on the last chunk is more idiomatic with WebAPI Streams, with the ability to **opt out of validation** for cases where it isn't needed.
## Related
A merkle inclusion proof API or header for byte range piece retrievals from Curio is also a TODO (client and server) if we're going to encourage or demonstrating byte range retrievals.
Contributor guide
Research direction
Start by tracing the download path and the accumulating reader in synapse-core, then compare it with the removed Retriever architecture. Decide how WebAPI Streams should report CommP validation, including the proposed opt-out, and verify the behavior for large downloads and the final validation result.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- api
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100