M2c.5: The physical operators the plan needs and exec does not have
- Dominant language
- Mojo
- Stars
- 1
- Forks
- 0
- Avg merge
- 1h 31m
- Merged PRs (30d)
- 640
Description
Part of #372. Depends on #376. The design is `docs/specs/planner/09-refactoring.md`.
`firepanda/exec/node.mojo` has `Filter`, `Project`, `Limit`, `Compute`, `Cast`, `Join`, `Group`, `Reduce` and a `Materialize` fallback. The logical node list in `docs/specs/planner/01-what-a-plan-is.md` has four things that set does not, and each of them is currently reachable only through `Materialize`, which is the whole column route wearing a node's clothes.
Each is one pull request and each removes one fallback. That incremental property, a plan that lowers what it can and materializes the rest, is what `docs/specs/planner/09-refactoring.md` calls the single most important structural idea to copy from Polars, and it is what lets this issue and M2b proceed without waiting on each other.
- [ ] `Sort`, consuming chunks and emitting them
- [ ] `Distinct`
- [ ] `Union`
- [ ] Semi and anti join kinds as first class nodes rather than through `Materialize`
## Quality
- [ ] Each new node has a test through `Pipeline`, in the shape `tests/test_group_node.mojo` already uses
- [ ] A test that a plan containing only lowerable nodes produces no `Materialize`
## Done when
The four nodes exist, and the TPC-H suite lowers with no `Materialize` in any of the twenty two plans.
Contributor guide
Research direction
Read docs/specs/planner/09-refactoring.md and docs/specs/planner/01-what-a-plan-is.md, then inspect the existing physical nodes in firepanda/exec/node.mojo. Use tests/test_group_node.mojo as the Pipeline test model and add coverage for each new node plus a plan containing only lowerable nodes. Done means the four operators are present and all twenty-two TPC-H plans lower without Materialize.
Written by the indexing model from the issue text.
Assessment
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 58/100