filecoin-project / filecoin-project/boost

Adding stream compression to retrievals

Open
#1,343 2 comments 1 reaction 1 assignee View on GitHub

@TippyFlitsUK is already working on this.

Since Apr 4, 2022.

area/markets area/markets/retrieval kind/feature need/analysis
Dominant language
Go
Stars
121
Forks
79
Avg merge
10d 14h
Merged PRs (30d)
1

Description

Checklist
  • This is not a new feature or an enhancement to the Filecoin protocol. If it is, please open an FIP issue.
  • This is not brainstorming ideas. If you have an idea you'd like to discuss, please open a new discussion on the lotus forum and select the category as Ideas.
  • I have a specific, actionable, and well motivated feature request to propose.
Lotus component
  • lotus daemon - chain sync
  • lotus miner - mining and block production
  • lotus miner/worker - sealing
  • lotus miner - proving(WindowPoSt)
  • lotus miner/market - storage deal
  • lotus miner/market - retrieval deal
  • lotus miner/market - data transfer
  • lotus client
  • lotus JSON-RPC API
  • lotus message management (mpool)
  • Other
What is the motivation behind this feature request? Is your feature request related to a problem? Please describe.

Knowing that there's quite some data that can very well be compressed (e.g. the Genome AD dataset) using tools like ZStandard (we've hit 90% compression on some of those .car files), bandwidth would greatly benefit from being able to compress the datastream coming out of retrieval deals.

Describe the solution you'd like

ZST could implemented as it supports streaming - if its used on both sides, a much higher throughput can be reached.

This will especially be useful as we're looking into the Evergreen project right now.

Describe alternatives you've considered

No response

Additional context

No response

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.