S5. The full SELECT surface, and the first conformance number
- Dominant language
- Mojo
- Stars
- 1
- Forks
- 0
- PR merge metrics
- PR metrics pending
Description
Part of #304. Depends on S4.
The rest of the SELECT surface, and the first published conformance number.
### Scope
- [ ] Window execution: partitioning, the three frame strategies, EXCLUDE and GROUPS framing
- [ ] Set operations including UNION BY NAME, and the by position and by name column matching rules
- [ ] DISTINCT and DISTINCT ON
- [ ] QUALIFY
- [ ] GROUPING SETS, CUBE, ROLLUP and the GROUPING() function reading the mask
- [ ] PIVOT and UNPIVOT
- [ ] The remaining join types: left, right, full, semi, anti, cross, positional, asof and lateral
- [ ] Recursive CTEs
- [ ] `unnest`, which is a table function in the grammar and a row multiplying plan node in practice
- [ ] The physical operators these need in `firepanda/exec/`: HashJoin, NestedLoopJoin, Sort, TopN, Limit, Window, Distinct, SetOp, Unnest, RecursiveCTE
- [ ] The conformance runner over DuckDB's `.test` files, and the first published per directory table
### Window strategies
Three, chosen per function. Naive per row for small frames and non decomposable aggregates. A segment tree over the partition for decomposable aggregates with arbitrary frames, which is DuckDB's approach and the only thing that makes RANGE frames tractable. And a streaming single state path for the running total case, ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW, which is by far the most common shape and is linear.
The ranking and navigation functions are not aggregates and get direct implementations over the sorted partition.
### The conformance rules
The `.test` files are run unmodified. No local edits, no patched expectations, no per file skips added to make a number look better. The corpus is vendored at the same DuckDB tag as the grammar and refreshed by the same procedure, because a corpus from one version checking a grammar from another produces failures that are nobody's bug.
Every failure is classified automatically into one of five buckets: an unsupported feature that refused by name, a missing function from tier two or three, a semantic divergence meaning right shape and wrong value or type, a crash, or a wrong answer with no error. The last two go to the top of the queue. The first two are the roadmap.
The number published is per directory with the denominator, because a single blended percentage over 4,046 files is not information. It is dominated by directories we deliberately have not implemented, and it can be moved by adding files rather than by adding features. Nothing is rounded up, and the README table is generated from the harness output rather than typed by a human, because a number a human types is a number that drifts.
### Exit criteria
- [ ] All 22 TPC-H queries return correct results at SF1
- [ ] The conformance rate over the target directories is published for the first time, per directory, with denominators
- [ ] Every corpus failure outside the target directories is classified, and none of them is a crash or a wrong answer
- [ ] The pass rate gains a ratchet in CI, so a commit that lowers it fails
### Depends on
S4.
Contributor guide
Research direction
Start by resolving S4, then inspect the physical operators in firepanda/exec/ and the DuckDB .test corpus used by the conformance runner. Run the target corpus and the 22 TPC-H queries at SF1; done means target-directory rates are published with denominators, failures are classified without crashes or wrong answers, and CI rejects a lower pass rate.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- sql
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 20/100