upload: interleave packing and uploading to bound staging disk
- Dominant language
- TypeScript
- Stars
- 1
- Forks
- 0
- Avg merge
- 5d 21h
- Merged PRs (30d)
- 1
Description
### Description
`upload` runs plan, then packs the entire CID list into CAR files, then starts uploading. Peak staging disk is therefore roughly the total size of the migration, even though CARs are evicted as copies commit during the upload phase.
### Impact
The runbook has to tell users to budget free disk equal to their data size. Issue #70 set out to keep the disk footprint near the pack target ("upload each ASAP, to keep disk small"), and #69 (hosted aggregator) assumes the local flow is disk-light for most users.
### Proposed next step
Interleave: build one bin, hand it to the upload loop, build the next while the upload streams. The commit batcher already tolerates pieces arriving over time, so only the pack scheduling moves.
Contributor guide
Research direction
Start by tracing the upload flow from plan through complete CAR packing and into the upload loop, then inspect how the commit batcher handles pieces arriving over time. Interleave bin construction with streaming uploads, and verify that staging disk stays near the pack target rather than the total migration size.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100