M10: Streaming and out of core
- Dominant language
- Mojo
- Stars
- 1
- Forks
- 0
- PR merge metrics
- PR metrics pending
Description
Tracking issue for **M10**. Specification: [`02-architecture.md`](../blob/main/docs/specs/02-architecture.md).
Polars took roughly three years to get here and shipped it incrementally behind per-operator fallback. Copy that approach.
### Scope
- [ ] A streamable declaration on the operator interface, with transparent fallback to the in-memory engine
- [ ] Spillable sinks for group by, join and sort, over a memory manager with a disk budget
- [ ] Sinks for Parquet, CSV, IPC and NDJSON that never materialize the whole result
- [ ] Partition-aware planning: prune Hive partitions, pre-partition group bys and joins on partition keys, rewrite inner joins as unions of partition-filtered joins
### Exit criteria
- [ ] db-benchmark at 50 GB completes within a memory budget smaller than the dataset
- [ ] Every operator either streams or falls back, with the choice visible in `explain()`
That second criterion is what makes incremental delivery honest. A user should never have to guess whether the thing they just ran streamed.
### Scheduling note
Highest effort, least parity value. **Ship 1.0 without it if anything is pressing** — nothing earlier depends on it, and the Hive partition metadata it needs has been carried on the scan node since M2 precisely so this can land late without a rewrite.
### Depends on
M5.
Contributor guide
Research direction
Start with docs/specs/02-architecture.md and review the dependency on M5 before breaking down the M10 scope. Done means operators stream or visibly fall back in explain(), sinks avoid whole-result materialization, and the 50 GB db-benchmark stays within a smaller memory budget.
Written by the indexing model from the issue text.
Assessment
- Domain
- data-engineering, performance, stream-processing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100