tamnd / tamnd/firepanda

M6b: ClickBench readiness

Open
#477 28 comments 0 reactions 0 assignees View on GitHub
area/benchmark area/frame area/string milestone performance
Dominant language
Mojo
Stars
1
Forks
0
Avg merge
1h 31m
Merged PRs (30d)
640

Description

This is the milestone. It is numbered M6b rather than getting a number of its own for the same reason M2b and M2c were: it slots in without renumbering anything after it. It sits next to M6 because the string namespace and the RE2 engine are its largest dependency, and it pulls forward the small parts of M8 that the suite actually needs while leaving the native Parquet reader where it is.

The suite side lives in tamnd/firepanda-bench#39. This issue is only the library work, which is the list of things firepanda cannot do that 43 published queries ask for.

## Why it exists

ClickBench is the suite an analytical engine is judged on in public, and it is the one that most resembles what a user has on disk. The three suites we run today are generated: db-benchmark comes out of a counter stream, TPC-H comes out of `dbgen`, the ingestion files are ours. All three have uniform key distributions, tidy types and no missing data, because that is what a generator produces. The hits table is a dump of what a real product recorded, with skewed cardinalities, empty strings standing in for nulls, and 105 columns of which most queries touch three.

Everything we have optimized has been optimized against uniform data. That is the gap this milestone is about.

There is a second reason and it is narrower. ClickBench has no joins in it at all. Every query is a scan, a filter, a group by and a sort over one table, which means a slow number cannot be blamed on join ordering. For a library that has just spent a milestone on a planner, a suite that cannot be rescued by better join order is a useful check on what the rest of the engine is worth.

## Where we stand before anything is written

Most of the suite runs today. Filters, group bys on one or two keys, the scalar aggregates, the sorts with a limit, `IN` lists, substring matching for `LIKE`: all present. The estimate is 31 of 43 on the first run, which is a better start than TPC-H had.

The twelve that do not run fall into six groups, and each is a sub issue below.

## The sub issues

Each is its own issue with its own checklist, and they are roughly in dependency order.

- [x] #478 M6b.0, the hundred and five column frame
- [x] #479 M6b.1, counting distinct values, once globally and once per group
- [x] #480 M6b.2, the three string operations the suite needs and one that waits on M6
- [x] #481 M6b.3, order by, then limit, then offset
- [x] #482 M6b.4, a conditional expression, and ninety sums in one pass
- [x] #483 M6b.5, truncating a timestamp to the minute, and reading the minute out of one
- [x] #484 M6b.6, a group by key that is almost unique at a hundred million rows
- [x] #485 M6b.7, the queries only a planner makes fast
- [x] #593 M6b.8, one hash of the tuple rather than one hash table per key

## What is explicitly not in here

**The native Parquet reader.** That is M8 and it stays there. Its absence means firepanda has no number at all at the published 100 million row size, because that size runs in scan mode where each engine reads the file itself, and firepanda reading a Parquet file means handing it to DuckDB. The suite publishes that as a refusal with the reason, the way it already does for TPC-H under scan. This milestone gets firepanda into the 1M and 10M tables, not into the published one.

**Approximate distinct counting.** ClickHouse answers several of these with a HyperLogLog estimate. We count exactly, everywhere, and we do not reach for an estimate to improve a number. If an approximate count is ever worth having it is worth having as a function a user asks for by name.

## Done when

firepanda has a cell in all 43 rows at 1M and 10M in memory mode, whether that cell is a number or a stated refusal, and every number agrees with DuckDB's answer. The only remaining refusals are the ones whose reason is the native Parquet reader.

Contributor guide

Open the contributing guide

Research direction

Start with subissues #478–#485 and the ClickBench suite in tamnd/firepanda-bench#39; they define the missing library capabilities and their dependency order. Compare the 43 published queries with DuckDB in memory mode, then verify that firepanda has a number or stated refusal for every 1M and 10M row case, with only native Parquet reader refusals remaining.

Written by the indexing model from the issue text.

Assessment

Domain
data-engineering, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.