tamnd / tamnd/firepanda

M4b: SQL, the DuckDB dialect

Open
#304 131 comments 0 reactions 0 assignees View on GitHub
area/plan area/sql enhancement milestone spec
Dominant language
Mojo
Stars
1
Forks
0
Avg merge
1h 31m
Merged PRs (30d)
640

Description

Tracking issue for **M4b**. Research and design: `docs/specs/sql/`, landing in the first pull request on this milestone.

DuckDB replaced its Bison parser with a hand written PEG parser in August 2026 and shipped the grammar as data. Forty `.gram` files, 1,087 rules, 61,190 bytes, MIT licensed, executed by the reference implementation itself. That is the whole reason this milestone moves ahead of M11. Eighteen months ago "compatible with DuckDB SQL" meant reading a Bison grammar and a hand written transformer and hoping. Today it means vendoring 61 KB of text and writing a matcher for it.

### Why this is M4b and not M11

`docs/specs/08-milestones.md` puts SQL last, on the path M4 to M6 to M7 to M11. Three things changed.

The parser stopped being the expensive part, for the reason above. M4 and this milestone are largely the same work, because M4 is the lazy engine, the expression nodes, the logical plan and the optimizer, and SQL binds to exactly those. Building them twice is the real risk of the current ordering. And SQL is the forcing function that makes the plan right, because the dialect is fixed by an external artifact and cannot be bent to suit a plan that cannot express correlated subqueries or grouping sets.

The counter argument is that this delays M6, which is pandas parity, which is the headline claim. If the signal after M3 says users want the pandas surface and do not care about SQL, this ordering is wrong and should be reverted. What does not change either way is that the plan built in M4 should be designed against DuckDB's dialect, because that costs nothing and prevents a rewrite.

### The stages

- [x] S0. Vendor the grammar and generate the rule table (#305)
- [ ] S1. Tokenizer and PEG matcher (#306)
- [ ] S2. AST, transformer and the refusal table (#307)
- [ ] S3. Binder, catalog, types and the tier one functions (#308)
- [ ] S4. The logical plan and the first executions (#309)
- [ ] S5. The full SELECT surface (#310)
- [ ] S6. The optimizer (#311)
- [ ] S7. The front doors (#312)
- [ ] S8. Memory and ADBC (#313)

### The four axes, stated so they can be falsified

**Dialect compatibility.** Every statement DuckDB accepts, firepanda accepts or refuses with an error that names the unimplemented feature. Never a syntax error for valid DuckDB SQL. Measured as the pass rate over DuckDB's own 4,046 `.test` files, reported per directory, published in the README, never rounded up.

**Query performance.** Twice DuckDB, Polars and pandas on every TPC-H query at SF10, or a written reason on the ones that are not. That is the bar #299 already set for the dataframe path. SQL adds no target of its own, because SQL that is slower than the dataframe call it lowers to is a bug in the binder.

**Latency.** `sql("SELECT a, sum(b) FROM t GROUP BY a")` over a registered frame of ten thousand rows, warm, returning a frame, under 100 microseconds end to end, of which under 20 is parse, bind, plan and optimize. A repeat execution of a statement whose text has been seen before under 2 microseconds to reach the physical plan. This is the axis where a multiple of ten is real, because the rivals are not slow, they are converting at both ends of a boundary we do not have.

**Memory.** Peak RSS within 1.2x of DuckDB on every TPC-H query at SF10, reported beside every timing rather than in a footnote. A query that does not fit in memory finishes rather than dies.

### What this is not

Not a database. No storage, no catalog persistence, no transactions, no ATTACH. Not a server. Not a SQL dialect of our own, because the moment we add one thing DuckDB does not have, the compatibility number stops meaning what it says. Not a wrapper around libduckdb, for the three reasons in the design docs, and not a fallback to it for the queries we cannot run, because a fallback would make the conformance number a measurement of DuckDB.

### Depends on

M4 for the logical plan and the optimizer passes. M2b for the chunked executor. Blocks nothing, and #13 loses the SQL scope if this lands.

Contributor guide

Open the contributing guide

Research direction

Start with docs/specs/sql/ and docs/specs/08-milestones.md, then read the linked stage issues, beginning with S1 (#306). Use DuckDB’s .test files and the four stated axes as the reference points; the milestone is done only when its stages and compatibility criteria are addressed.

Written by the indexing model from the issue text.

Assessment

Tech stack
sql
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.