tamnd / tamnd/firepanda

M4: Lazy engine, expressions and optimizer

Open
#6 0 comments 0 reactions 0 assignees View on GitHub
area/expr area/plan milestone
Dominant language
Mojo
Stars
1
Forks
0
Avg merge
1h 31m
Merged PRs (30d)
640

Description

Tracking issue for **M4**. Specification: [`02-architecture.md`](../blob/main/docs/specs/02-architecture.md), [`04-python-dx.md`](../blob/main/docs/specs/04-python-dx.md).

The milestone that defines the product, and the most complex one.

### Scope

- [ ] Expression node types, immutable and structurally shared so that common subexpression elimination is a pointer comparison
- [ ] `LazyFrame` with the operations from document 02
- [ ] The logical plan, plan-time type checking and validation
- [ ] Optimizer passes: projection pushdown, predicate pushdown, slice pushdown, CSE, constant folding, type-coercion insertion, dictionary preservation, expression fusion
- [ ] **Rewire the eager surface to build plans instead of materializing**
- [ ] `explain()` and `profile()` — here, not later
- [ ] The error model: poisoned nodes, did-you-mean suggestions, plan position in the message

### The highest-leverage item

Rewiring the eager surface is what makes the naive pandas idiom fast. A user who writes `df[df.qty > 1000].groupby("sym").sum()` gets projection and predicate pushdown without learning an expression API. It also means every M1 test now exercises the lazy path, which is the differential coverage that makes the optimizer safe.

### Exit criteria

- [ ] Every optimizer pass has a test asserting on `explain()` output — the plan shape is the assertion
- [ ] Pushdown is verified by instrumented counters for columns decoded and row groups read, **not by timing**
- [ ] Every eager operation from M1 produces identical results through the new deferred path, checked by the M1 suite unchanged
- [ ] `__repr__` on an unmaterialized frame over 100M rows returns in under 200ms

### The risk

A wrong query plan still returns a plausible-looking result. The defence is that the M1 test suite becomes the differential oracle — which is only true if it was written to be one.

### Depends on

M2. Runs independently of M3.

Contributor guide

Open the contributing guide

Research direction

Start with docs/specs/02-architecture.md and docs/specs/04-python-dx.md, then review the M2 dependency and unchanged M1 suite before selecting a narrowly scoped component. Done requires tests for the relevant explain() plan shape, instrumented pushdown counters where applicable, and unchanged M1 results through the deferred path.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
backend-api-design, data-engineering, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
15/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.