tamnd / tamnd/firepanda

S6. The optimizer

Open
#311 0 comments 0 reactions 0 assignees View on GitHub
area/plan area/sql performance
Dominant language
Mojo
Stars
1
Forks
0
Avg merge
1h 31m
Merged PRs (30d)
640

Description

Part of #304. Depends on S5, and shares its pass list with M4.

Plan in, better plan out, same answer. This is where the query performance axis is actually won or lost, because on TPC-H the difference between a good plan and a naive one is two orders of magnitude and no amount of kernel tuning closes it.

It is also where a project like this most often overreaches, so the scope is ordered by return on effort and it names what waits.

### Scope

- [ ] The pass pipeline, a fixed sequence, each pass individually disableable by name from a setting
- [ ] Expression rewriting: constant folding, comparison simplification, CASE elimination, IN lowering, LIKE lowering, common subexpression elimination
- [ ] Filter pushdown with predicate inference across equi joins and null rejecting predicate detection
- [ ] Projection pushdown to the scans
- [ ] Statistics propagation from the scans upward
- [ ] Join order by dynamic programming over the join graph, with a greedy fallback above a vertex threshold
- [ ] Build side selection
- [ ] Limit pushdown, and the TopN rewrite
- [ ] Late materialization
- [ ] The optimizer off equivalence test and the plan stability snapshots

### Not a cost based rewrite search

A fixed pass list, not a Cascades style memo. DuckDB's optimizer is a pass list and it is competitive with anything, while a memo framework is a year of work whose payoff arrives only after the pass list is exhausted.

Every pass takes a plan and returns a plan, every pass is testable against a written input and expected output in the JSON form from S4, and every pass can be turned off by name. That last part is not a convenience. It is how a wrong answer gets bisected to a pass in one minute rather than one day, and DuckDB has exactly this for exactly that reason.

### Pushdown is most of the win

Filter pushdown moves predicates toward the scans, through projections, through joins on the appropriate side, and into the scan's filter slot. On a Parquet scan it becomes a row group skip, which is the difference between reading the file and not.

Three refinements matter, in order. Predicate inference across equi joins, so `a.x = b.x AND a.x > 5` implies `b.x > 5` and the derived predicate pushes to the other side. On TPC-H this is worth more than any other single rewrite, because it turns a filter on one table into a filter on both. Conjunction splitting before pushdown, so a predicate is pushed conjunct by conjunct rather than getting stuck at the first node that cannot take all of it. And null rejecting predicate detection, which converts an outer join to an inner join when a predicate above it would discard the null extended rows, unlocking reordering that is otherwise blocked.

### Two rules on the expression rewriter

It may never change a type. A rewrite that produces a different `typeof()` is a compatibility failure. Folding `1.1 + 2.2` has to produce `DECIMAL(3,1)` and `3.3`, not a double.

It may never turn a raising expression into a non raising one or the reverse. Folding `1/0` at plan time when the branch would never have executed is a wrong answer, so folding stops at anything that can raise under the S3 overflow rules unless the operands are known safe.

### Statistics, and the honest limitation

Exact row counts, because our inputs are in memory frames and we know, which is a real advantage over a database estimating from a stale sample. Per column min, max and null count. HyperLogLog distinct counts for join key selectivity. Fixed selectivity constants for everything else, which is what every engine actually does for non equality predicates.

Correlated predicates are estimated as independent and the estimate is therefore wrong, sometimes by orders of magnitude. That is enough for TPC-H and not enough in general, and the trigger for building sketches is a specific one: a benchmark query picking a plan more than twice worse than the best available, where the cause traces to an estimate rather than to a missing rewrite.

### Late materialization

The pass with the best ratio of value to complexity for a dataframe engine specifically, and Polars and DuckDB arrived at it independently.

Sort, join and top N on a wide frame move payload columns they never inspect. The rewrite carries a row identifier through the operator and joins the payload back afterwards. On `ORDER BY x LIMIT 10` over a fifty column frame that is the difference between sorting fifty columns and sorting one plus a gather of ten rows. Our version is stronger than a database's, because our inputs are already in memory and already columnar, so gathering the payload later is a real gather against real memory rather than a re-read.

### What waits

Sketch based cost estimation until the trigger above fires. Subquery caching and materialized CTE cost decisions, since inline by default plus honouring the explicit hints is enough until a benchmark says otherwise. Adaptive conjunct reordering at run time, which belongs in the expression executor rather than here. Semi join reduction, bloom filters and sideways information passing, which are real wins on star schemas and none of which matters until the list above is done and measured.

### Exit criteria

- [ ] All 22 TPC-H queries complete at SF10
- [ ] Optimizer off and optimizer on produce identical results across the corpus, including row order where the query specifies one
- [ ] The 22 plan snapshots are in the repository, and a pass change that moves one has to say why in the commit
- [ ] Every pass has a unit test written as an input plan and an expected output plan

### Depends on

S5.

Contributor guide

Open the contributing guide

Research direction

Start with S5 and the JSON plan representation from S4, then map the proposed fixed pass pipeline and its tests. The scope includes expression rewriting, pushdown, statistics, join ordering, and late materialization, so confirm the existing plan and scan entry points before proceeding. Done requires the 22 TPC-H queries, optimizer-on/off result equivalence, 22 plan snapshots, and input/output tests for every pass.

Written by the indexing model from the issue text.

Assessment

Domain
data-engineering, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.