tamnd / tamnd/firepanda

TPC-H, all twenty two queries, and the operations they need that we do not have

Open
#299 10 comments 0 reactions 0 assignees View on GitHub
area/benchmark area/io area/kernel area/string enhancement performance
Dominant language
Mojo
Stars
1
Forks
0
PR merge metrics
PR metrics pending

Description

The benchmark repository has run TPC-H since it was set up, and firepanda has never appeared in the table. Every one of the twenty two rows carries an explicit refusal instead of a number. That is honest, and it has also been true long enough that it is now the largest hole in the comparison: db-benchmark is nine group by queries and six joins on synthetic data, and TPC-H is the suite that asks whether an engine can run a query somebody actually wrote.

This issue is the plan to close it, and the goal is the same one as everywhere else on the performance side: twice DuckDB, Polars and pandas on every query, at the sizes the report leads with.

### Why the refusal exists today

`tools/engines/firepanda_engine.py` says it plainly. firepanda has no Parquet decoder of its own, it opens a Parquet file by handing it to DuckDB and reading DuckDB's vectors back as Arrow, and DuckDB is one of the four engines in the table. For db-benchmark the driver works around that by generating the same data from the same splitmix64 stream, and the cross engine fingerprint is what makes that claim testable. TPC-H data comes from dbgen and there is no seed to reproduce, so that workaround does not transfer.

The second reason is the queries themselves. Once the data is in, the twenty two need several operations firepanda does not have. That half is worth doing regardless of the reader, because those operations are missing from a dataframe library and not only from a benchmark.

### The two phases, and why in this order

Phase one runs TPC-H in the harness's default `--io memory` mode, where every engine is handed materialized tables and the read is outside the timed region. Polars loads eagerly there rather than scanning, pandas has no scan at all, and DuckDB registers Arrow views. In that mode firepanda loading through the DuckDB bridge is outside the timed region too, and the query comparison is between engines rather than between readers. This gets twenty two real numbers as soon as the query operations land, and the load column says exactly how firepanda's tables arrived.

Phase two is the native Parquet reader, which is what makes `--io scan` meaningful and removes the caveat entirely. It is also an M2 exit criterion already, and is tracked at #211 for the Python extension side.

Doing them in this order means the query work is not blocked behind a decoder, and the decoder is not being written to a schedule set by a benchmark.

### Scope: the operations the queries need and we do not have

- [x] `contains` on a text column, which is the `LIKE '%x%'` in q9, q13 and q16
- [x] `starts_with` on a text column, which is `LIKE 'x%'` in q14, q20 and q22
- [x] `ends_with` on a text column, which is `LIKE '%x'` in q2
- [x] A two part pattern, `LIKE '%a%b%'`, which q13 needs and which a single substring search does not answer
- [x] `substring` by byte range on a text column, which q22 takes the first two characters of a phone number with
- [x] `is_in` against a small set of values, integer and text, which q19 and q22 need and which is a join wearing an inconvenient shape today
- [x] A conditional column, the `CASE WHEN` in q8, q12 and q14, which at the frame level is a mask and two columns
- [x] A distinct count aggregate, which q16 asks for per group, already present as `AggKind.NUNIQUE` before this issue was written
- [x] Distinct rows over a frame, which several of the decorrelations need and which only `Index.unique` can do today
- [x] A grouped filter, so a group by can be followed by a predicate on the aggregate without materializing and rejoining

### Scope: the queries in the driver

Each of these is the official statement expressed as dataframe operations, with the specification's substitution parameters, checked against `tpch_answers()` before it counts.

- [x] q1, pricing summary report, one table and one group by
- [x] q2, minimum cost supplier, a correlated subquery over five tables decorrelated into a group by and a join
- [x] q3, shipping priority, three tables and a top ten
- [x] q4, order priority checking, a semi join
- [x] q5, local supplier volume, six tables
- [x] q6, forecasting revenue change, one table and three predicates
- [x] q7, volume shipping, a self join on nation
- [x] q8, national market share, eight tables and a conditional
- [x] q9, product type profit measure, a pattern match and a year extraction
- [x] q10, returned item reporting, four tables and a top twenty
- [x] q11, important stock identification, a group by compared against a scalar subquery
- [x] q12, shipping modes and order priority, two conditionals
- [x] q13, customer distribution, a left join and a group by of a group by
- [x] q14, promotion effect, a conditional over a pattern match
- [x] q15, top supplier, a view defined as a group by and joined against its own maximum
- [x] q16, parts supplier relationship, an anti join and a distinct count
- [x] q17, small quantity order revenue, a correlated subquery on the average
- [x] q18, large volume customer, a group by used as a filter on another
- [x] q19, discounted revenue, a disjunction of three predicate groups
- [x] q20, potential part promotion, two nested subqueries and a pattern match
- [x] q21, suppliers who kept orders waiting, a semi join and an anti join on the same table
- [x] q22, global sales opportunity, a substring, an `is_in` and a scalar subquery

### Scope: the reader, which is phase two

- [ ] Read an uncompressed Parquet file's footer and schema without DuckDB
- [ ] PLAIN and RLE dictionary encodings for the physical types dbgen produces
- [ ] Decimal to double, which is what the other three engines do with these columns
- [ ] Date and string columns
- [ ] Row group and column projection, so `--io scan` measures what it claims to
- [ ] Within twice Polars on the same file, which is the M2 exit criterion

### Scope: honesty and validation

- [x] The refusal strings in the bench engine are replaced by numbers or by a reason that names the missing operation, which is tamnd/firepanda-bench#22
- [x] Every query is checked against `tpch_answers()` at sf1 before it is allowed into a result file
- [ ] The report says which engines scanned and which were handed materialized tables, per suite, rather than in a footnote
- [ ] firepanda's load time stays in its own column, because generating or bridging is not reading

### Exit criteria

- [x] All twenty two queries run and reproduce the specification's answers at sf1
- [x] A four engine table at sf1 and at sf10 on the 13900K, with peak memory beside every time
- [ ] Twice DuckDB, Polars and pandas on every query, or a written reason on each one that is not
- [ ] The native reader lands and the scan mode table is published beside the memory mode one

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.