Five TPC-H queries are refused, and they stop in four places
- Dominant language
- Mojo
- Stars
- 1
- Forks
- 0
- Avg merge
- 1h 31m
- Merged PRs (30d)
- 640
Description
`pixi run tpch` asks whole TPC-H queries of firepanda and of DuckDB over the same Parquet files and compares the answers row for row. Ten of the twenty two were in its list when this was filed, and asking all twenty two said where the other twelve stopped: five places rather than twelve.
Three of those five are closed. What is left is five queries stopping in four places:
```
17 of 22 queries agree with DuckDB over the same rows
```
This issue is the list, so that the harness can record a reason against every refusal and notice the day one of them closes. Every number below is at scale factor 0.01, which is what the harness runs by default.
## Closed: a decimal literal, which was stopping five queries
q6, q14, q17, q20 and q22 were refused with this:
```
firepanda does not lower the decimal literal 0.05 yet, because a plan cannot hold an exact decimal and a double in its place would answer 1.1 + 2.2 with 3.3000000000000003
```
The refusal was right about the risk and refusing more than the risk. In all five the literal meets a double, and DuckDB casts the decimal to a double in every one of them, so a double in its place was the same answer and not a wrong one. The fix in #825 is a rule about where the literal sits rather than a decimal type: an expression of decimal literals folds exactly, in base ten, and is converted once at the boundary where it meets something that is not a decimal. `1.1 + 2.2` is still exact, and `l_discount BETWEEN 0.05 AND 0.07` gets the two doubles DuckDB has.
q6, q14 and q22 agree with DuckDB now. q17 and q20 moved past this.
## Closed: a cross join with more than one row on the right, which was stopping four
q7, q8, q9 and q19. Two separate things were behind one message and neither turned out to need the operator.
q7 was a predicate placed by column name alone, fixed in #823. q19 was a conjunct written into every branch of an `OR` rather than once outside it, hoisted in #819. q8 and q9 both write the table all their equalities are against third in a list of six or eight, so the first join in the chain crossed everything, and #831 orders a comma `FROM` so that no pair of relations is left crossed.
All four agree with DuckDB. There is still no operator that pairs every left row with every right row, and nothing in TPC-H now needs one.
## Closed: a correlation written with no qualifier in front of it
```
there is no column named 'p_partkey' here. Did you mean 'ps_partkey'?
```
q2, q17 and q20 each write a subquery whose `WHERE` reads a column of the query around it without a table name in front of it, and the test for whether a subquery is correlated only answered a qualified name. #832 asks the catalog for the bare half, so a bare name that no table in the subquery has and some table outside it does is recognised as the correlation it is.
All three are past this and none of them answers yet. They stop in the two places below.
## Closed: a table named on both sides of a correlation
```
'partsupp' is the name of more than one table in this FROM
```
`_folded_join` lowers the subquery's `FROM` into the scope the outer query is using, which is what puts both sides of the correlation in reach at once so a condition reading both can be written. A table the subquery names that the query around it also names then collided in that one scope. #841 draws a line in the scope marking where the innermost query's names begin, refuses a repeat above it and allows one across it, looks a relation name up from the end, and pins a bare column written under the line to the innermost relation that has it.
q17 answers on that together with #839, which made a sum over no rows null rather than zero. q2 and q20 are past it and stop on the two operator limits below.
## A cross join with more than one row on the right, which stops q2
```
lower: a cross join pairs every left row with every right row, and this one has a right side of 100 rows, which is the whole frame join rather than a column added as each chunk goes past. One right row is the case that lowers
```
This was on the list once already, as the thing that looked like it was stopping q7, q8, q9 and q19. None of those four needed it in the end. q2 does.
## A left join on two key pairs, which stops q20
q20 stops on a left join whose condition is two equalities rather than one, which wants the ordinal space that concatenating both key columns builds. An operator limit and nothing to do with resolution.
## A scalar subquery in a place the lowering does not take, which stops q11
```
firepanda lowers an uncorrelated subquery that answers one value where it is written in a WHERE, or in the select list of a query that does not aggregate, and this one is written somewhere else. The answer is a column cross joined on above the FROM, which is under the aggregate, and an aggregate hands up its keys and its folds rather than everything it read
```
q11 compares a group's total against a scalar subquery in a `HAVING`, which is above the aggregate rather than below it.
## A join condition that is not an equality, which stops q13 and q21
```
firepanda lowers a left join on equalities between its two sides so far, and the rest of this condition decides which rows are kept rather than which rows match, so it cannot be tested above the join instead
```
```
firepanda decorrelates a EXISTS whose subquery reads the query around it through equalities and nothing else, and this part of its condition reads it another way, which is the dependent join that a decorrelation pass removes rather than one this rewrite can
```
q13 has `o_comment NOT LIKE '%special%requests%'` in its `LEFT JOIN ... ON`, where a predicate above the join would drop the padded rows and change the answer. q21 correlates its `EXISTS` through an inequality on `l_suppkey` as well as through an equality.
## Checklist
- [x] A decimal literal that meets something which is not a decimal literal is read as a double, and an expression of decimal literals alone stays refused
- [x] q6, q14 and q22 agree with DuckDB and come out of the recorded list
- [x] A conjunct common to every branch of a disjunction is hoisted out of it, so q19 joins where it used to cross
- [x] q19 agrees with DuckDB
- [x] A comma `FROM` is ordered so that no pair of relations is left crossed, so q8 and q9 join where they used to cross
- [x] q7, q8 and q9 agree with DuckDB
- [x] A correlation written with no qualifier in front of it is recognised as one, so q2, q17 and q20 get past the resolution
- [x] A table named on both sides of a correlation shadows rather than collides, so q17 agrees and q2 gets past the names
- [ ] A left join on more than one key pair, so q20 agrees
- [ ] A scalar subquery is taken in a `HAVING`, so q11 agrees
- [ ] A left join keeps a condition that is not an equality, so q13 agrees
- [ ] An `EXISTS` correlated by more than equalities decorrelates, so q21 agrees
- [ ] An operator that pairs every left row with every right row, so a cross join is not limited to a right side of one row and q2 agrees
- [ ] `pixi run tpch` says 22 of 22 and `recorded` is empty
Contributor guide
Research direction
Start by running `pixi run tpch` and inspect the recorded refusals for q2, q11, q13, q20, and q21. The checklist defines done: each remaining limitation is addressed, the affected queries agree with DuckDB, and `pixi run tpch` reports 22 of 22 with `recorded` empty; no source files or tests are named.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- sql
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100