FilOzone / FilOzone/synapse-sdk

Support larger files than 1GiB

Open
#110 15 comments 2 reactions 1 assignee Claimed by @rvagg View on GitHub
design
Dominant language
TypeScript
Stars
25
Forks
32
Avg merge
3d 4h
Merged PRs (30d)
22

Description

Let's consider this scenario:
1. I have a 2 GB video file I want to stream from Filecoin.
2. I upload this file to PDP.
3. In my video player widget, I want to fetch a byte range of my 2 GB file to start playing the content quickly.
4. Ideally, I also want to verify that the byte range returned by the server belongs to my file. (An inclusion proof.)

As a builder:
- The fact that PDP pieces are limited to 254MiB is an implementation detail of PDP & Synapse that I don't want to deal with.
- I want to give Synapse SDK a ReadableStream with my 2 GB video file and get back a single CID I can use for retrieving my file later. (My mental model is Object Storage - I upload a blob of data and get back a key I can use to retrieve my blob.)
- In my video player, I want to use that single CID to identify the video to play, and fetch chunks as a byte range based on the size of the entire file.
- https://github.com/FilOzone/fs-pm/issues/115

**The goal of this issue is to discuss and implement an easy-to-use end-to-end solution.**

Potential building blocks:
- https://github.com/filecoin-project/curio/issues/541
- https://github.com/FilOzone/synapse-sdk/issues/51

Notes:
- Content verification for byte-range retrievals is tricky. Some options that come to my mind:
- Adapt [Blake3's BAO format](https://github.com/n0-computer/bao-docs) for CommP.
- Commpact https://www.youtube.com/watch?v=ndLKjXQ1awI
- To make uploading large files practical, we need to implement upload resumption.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.