M4: Lazy engine, expressions and optimizer
- Dominant language
- Mojo
- Stars
- 1
- Forks
- 0
- Avg merge
- 1h 31m
- Merged PRs (30d)
- 640
Description
Tracking issue for **M4**. Specification: [`02-architecture.md`](../blob/main/docs/specs/02-architecture.md), [`04-python-dx.md`](../blob/main/docs/specs/04-python-dx.md).
The milestone that defines the product, and the most complex one.
### Scope
- [ ] Expression node types, immutable and structurally shared so that common subexpression elimination is a pointer comparison
- [ ] `LazyFrame` with the operations from document 02
- [ ] The logical plan, plan-time type checking and validation
- [ ] Optimizer passes: projection pushdown, predicate pushdown, slice pushdown, CSE, constant folding, type-coercion insertion, dictionary preservation, expression fusion
- [ ] **Rewire the eager surface to build plans instead of materializing**
- [ ] `explain()` and `profile()` — here, not later
- [ ] The error model: poisoned nodes, did-you-mean suggestions, plan position in the message
### The highest-leverage item
Rewiring the eager surface is what makes the naive pandas idiom fast. A user who writes `df[df.qty > 1000].groupby("sym").sum()` gets projection and predicate pushdown without learning an expression API. It also means every M1 test now exercises the lazy path, which is the differential coverage that makes the optimizer safe.
### Exit criteria
- [ ] Every optimizer pass has a test asserting on `explain()` output — the plan shape is the assertion
- [ ] Pushdown is verified by instrumented counters for columns decoded and row groups read, **not by timing**
- [ ] Every eager operation from M1 produces identical results through the new deferred path, checked by the M1 suite unchanged
- [ ] `__repr__` on an unmaterialized frame over 100M rows returns in under 200ms
### The risk
A wrong query plan still returns a plausible-looking result. The defence is that the M1 test suite becomes the differential oracle — which is only true if it was written to be one.
### Depends on
M2. Runs independently of M3.
Contributor guide
Research direction
Start with docs/specs/02-architecture.md and docs/specs/04-python-dx.md, then review the M2 dependency and unchanged M1 suite before selecting a narrowly scoped component. Done requires tests for the relevant explain() plan shape, instrumented pushdown counters where applicable, and unchanged M1 results through the deferred path.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- pandas, python
- Domain
- backend-api-design, data-engineering, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 15/100