filecoin-project / filecoin-project/filecoin-pin

feat: support batching adds to amortize addPieces fees

Open
#692 0 comments 0 reactions 0 assignees View on GitHub
team/filecoin-pin team/fs-wg
Dominant language
TypeScript
Stars
26
Forks
21
Avg merge
5d 18h
Merged PRs (30d)
27

Description

`filecoin-pin add` commits one piece per invocation: `uploadToSynapse` calls `synapse.storage.upload()` once per CAR (`src/core/upload/synapse.ts:355`), which stores and commits in the same call. We should support batching adds, packing many items into fewer pieces and committing them in one `addPieces` transaction.

`addPieces` is charged per transaction, so adding a pile of small pieces one at a time can cost users a couple hundred dollars where the same content packed into one larger piece costs cents.

`add ` already covers content that is co-located, since it walks the tree into one CAR and one piece. The gap is items that arrive over time or from separate sources.

The migrate stack (#652 through #657) already does packing and batched commits for content fetched from IPFS gateways, so `pack-cars.ts` and `gc-window.ts` are the parts to build on.

Related: #651, #190.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.