tamnd / tamnd/firepanda

M10: Streaming and out of core

Open
#12 0 comments 0 reactions 0 assignees View on GitHub
area/exec area/streaming milestone
Dominant language
Mojo
Stars
1
Forks
0
PR merge metrics
PR metrics pending

Description

Tracking issue for **M10**. Specification: [`02-architecture.md`](../blob/main/docs/specs/02-architecture.md).

Polars took roughly three years to get here and shipped it incrementally behind per-operator fallback. Copy that approach.

### Scope

- [ ] A streamable declaration on the operator interface, with transparent fallback to the in-memory engine
- [ ] Spillable sinks for group by, join and sort, over a memory manager with a disk budget
- [ ] Sinks for Parquet, CSV, IPC and NDJSON that never materialize the whole result
- [ ] Partition-aware planning: prune Hive partitions, pre-partition group bys and joins on partition keys, rewrite inner joins as unions of partition-filtered joins

### Exit criteria

- [ ] db-benchmark at 50 GB completes within a memory budget smaller than the dataset
- [ ] Every operator either streams or falls back, with the choice visible in `explain()`

That second criterion is what makes incremental delivery honest. A user should never have to guess whether the thing they just ran streamed.

### Scheduling note

Highest effort, least parity value. **Ship 1.0 without it if anything is pressing** — nothing earlier depends on it, and the Hive partition metadata it needs has been carried on the scan node since M2 precisely so this can land late without a rewrite.

### Depends on

M5.

Contributor guide

Open the contributing guide

Research direction

Start with docs/specs/02-architecture.md and review the dependency on M5 before breaking down the M10 scope. Done means operators stream or visibly fall back in explain(), sinks avoid whole-result materialization, and the 50 GB db-benchmark stays within a smaller memory budget.

Written by the indexing model from the issue text.

Assessment

Domain
data-engineering, performance, stream-processing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.