FilOzone / FilOzone/ipfs2foc

upload: interleave packing and uploading to bound staging disk

Open
#72 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
1
Forks
0
Avg merge
5d 21h
Merged PRs (30d)
1

Description

### Description

`upload` runs plan, then packs the entire CID list into CAR files, then starts uploading. Peak staging disk is therefore roughly the total size of the migration, even though CARs are evicted as copies commit during the upload phase.

### Impact

The runbook has to tell users to budget free disk equal to their data size. Issue #70 set out to keep the disk footprint near the pack target ("upload each ASAP, to keep disk small"), and #69 (hosted aggregator) assumes the local flow is disk-light for most users.

### Proposed next step

Interleave: build one bin, hand it to the upload loop, build the next while the upload streams. The commit batcher already tolerates pieces arriving over time, so only the pack scheduling moves.

Contributor guide

Open the contributing guide

Research direction

Start by tracing the upload flow from plan through complete CAR packing and into the upload loop, then inspect how the commit batcher handles pieces arriving over time. Interleave bin construction with streaming uploads, and verify that staging disk stays near the pack target rather than the total migration size.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.