FilOzone / FilOzone/synapse-sdk
Support larger files than 1GiB
- Dominant language
- TypeScript
- Stars
- 25
- Forks
- 32
- Avg merge
- 3d 4h
- Merged PRs (30d)
- 22
Description
Let's consider this scenario:
1. I have a 2 GB video file I want to stream from Filecoin.
2. I upload this file to PDP.
3. In my video player widget, I want to fetch a byte range of my 2 GB file to start playing the content quickly.
4. Ideally, I also want to verify that the byte range returned by the server belongs to my file. (An inclusion proof.)
As a builder:
- The fact that PDP pieces are limited to 254MiB is an implementation detail of PDP & Synapse that I don't want to deal with.
- I want to give Synapse SDK a ReadableStream with my 2 GB video file and get back a single CID I can use for retrieving my file later. (My mental model is Object Storage - I upload a blob of data and get back a key I can use to retrieve my blob.)
- In my video player, I want to use that single CID to identify the video to play, and fetch chunks as a byte range based on the size of the entire file.
- https://github.com/FilOzone/fs-pm/issues/115
**The goal of this issue is to discuss and implement an easy-to-use end-to-end solution.**
Potential building blocks:
- https://github.com/filecoin-project/curio/issues/541
- https://github.com/FilOzone/synapse-sdk/issues/51
Notes:
- Content verification for byte-range retrievals is tricky. Some options that come to my mind:
- Adapt [Blake3's BAO format](https://github.com/n0-computer/bao-docs) for CommP.
- Commpact https://www.youtube.com/watch?v=ndLKjXQ1awI
- To make uploading large files practical, we need to implement upload resumption.
Contributor guide
Assessment
This issue has not been assessed yet.