M2c.1: The plan layer, the IR and binding
- Dominant language
- Mojo
- Stars
- 1
- Forks
- 0
- PR merge metrics
- PR metrics pending
Description
Part of #372. The design is `docs/specs/planner/01-what-a-plan-is.md` and `docs/specs/planner/09-refactoring.md`.
This is the milestone inside the milestone. At the end of it the chunked engine in `firepanda/exec/` is reachable from the public API, which it is not today: it has nine physical operators and a pipeline driver and the only thing in the repository that calls it is `tests/test_group_node.mojo`.
The new package is `firepanda/plan/`. `node.mojo` for the logical node kinds, `expr.mojo` for the expression kinds and the analyses, `bind.mojo` for name and type resolution, `pass/` for the rewrites one file per pass, `lower.mojo` for turning a logical plan into the existing `exec` nodes, and `print.mojo` for the two explain forms. The name collision between `plan/node.mojo` and `exec/node.mojo` is the right collision, because they are the same concept at two levels.
## The nodes
- [x] `Scan`
- [x] `Filter`
- [x] `Project`
- [x] `Aggregate`
- [x] `Join`
- [x] `Sort`
- [x] `Limit`
- [x] `Distinct`
- [x] `Union`
## The expressions
- [x] Column reference
- [x] Literal
- [x] Unary
- [x] Binary
- [x] Cast
- [x] Call
- [x] Aggregate
- [x] Conditional
- [x] Window
## Binding
Names to positions and types, once. This is what retires `DataFrame.column`'s copy as a problem rather than working around it, which every helper in the TPC-H benchmark driver currently does by hand.
- [x] Name to position resolution against the input schema
- [x] Logical type computed at bind time, so a type error is a plan error rather than a kernel error
- [x] A did you mean suggestion on an unresolved name
## The analyses
Every pass after this is written in terms of these three, so they come before any pass does.
- [x] Elementwise: whether an expression's output row depends only on its own input row
- [x] Input independent: whether an expression can be evaluated without reading any row
- [x] Table set: which scans an expression reads from
## Lowering and printing
- [x] Lowering into the existing `exec` node union
- [x] Anything not yet lowerable refused by name, which is what keeps this issue and M2b independent of each other. Not `Materialize`, and the comment below says why
- [ ] `explain(optimized=False)` printing the unoptimized form
- [ ] `explain()` printing the optimized form
- [ ] The eager `DataFrame` methods keeping their signatures and their behaviour, each building a one node plan over an in memory scan and collecting it immediately
## Quality
- [x] Plan tests written as a plan in and a printed plan out, compared as text
- [ ] A differential test running the existing suite through the plan path and the direct path and comparing answers
## Done when
The physical engine is reachable from the public API, every existing test passes unchanged, and `explain()` prints something a person can read.
Contributor guide
Research direction
Read docs/specs/planner/01-what-a-plan-is.md and docs/specs/planner/09-refactoring.md, then inspect firepanda/exec/ and tests/test_group_node.mojo to understand the existing physical engine. The remaining scope is in firepanda/plan/, including explain printing, eager DataFrame methods, and the differential test. Done means the public API reaches the physical engine, existing tests remain unchanged and passing, and explain() produces readable output.
Written by the indexing model from the issue text.
Assessment
- Domain
- backend-api-design, data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100