tamnd / tamnd/firepanda

M2c.5: The physical operators the plan needs and exec does not have

Open
#380 0 comments 0 reactions 0 assignees View on GitHub
area/exec area/plan milestone
Dominant language
Mojo
Stars
1
Forks
0
Avg merge
1h 31m
Merged PRs (30d)
640

Description

Part of #372. Depends on #376. The design is `docs/specs/planner/09-refactoring.md`.

`firepanda/exec/node.mojo` has `Filter`, `Project`, `Limit`, `Compute`, `Cast`, `Join`, `Group`, `Reduce` and a `Materialize` fallback. The logical node list in `docs/specs/planner/01-what-a-plan-is.md` has four things that set does not, and each of them is currently reachable only through `Materialize`, which is the whole column route wearing a node's clothes.

Each is one pull request and each removes one fallback. That incremental property, a plan that lowers what it can and materializes the rest, is what `docs/specs/planner/09-refactoring.md` calls the single most important structural idea to copy from Polars, and it is what lets this issue and M2b proceed without waiting on each other.

- [ ] `Sort`, consuming chunks and emitting them
- [ ] `Distinct`
- [ ] `Union`
- [ ] Semi and anti join kinds as first class nodes rather than through `Materialize`

## Quality

- [ ] Each new node has a test through `Pipeline`, in the shape `tests/test_group_node.mojo` already uses
- [ ] A test that a plan containing only lowerable nodes produces no `Materialize`

## Done when

The four nodes exist, and the TPC-H suite lowers with no `Materialize` in any of the twenty two plans.

Contributor guide

Open the contributing guide

Research direction

Read docs/specs/planner/09-refactoring.md and docs/specs/planner/01-what-a-plan-is.md, then inspect the existing physical nodes in firepanda/exec/node.mojo. Use tests/test_group_node.mojo as the Pipeline test model and add coverage for each new node plus a plan containing only lowerable nodes. Done means the four operators are present and all twenty-two TPC-H plans lower without Materialize.

Written by the indexing model from the issue text.

Assessment

Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.