tamnd / tamnd/firepanda

S3. The binder, the catalog, the types and the tier one functions

Open
#308 5 comments 0 reactions 0 assignees View on GitHub
area/dtype area/sql enhancement test
Dominant language
Mojo
Stars
1
Forks
0
Avg merge
1h 31m
Merged PRs (30d)
640

Description

Part of #304. Depends on S2.

Name resolution, the type system and the first tier of functions. This is where most of DuckDB's behaviour that is not in the grammar actually lives, and it is the issue most likely to be underestimated, because the parser is what people think the hard part is and it is not.

### Scope

- [x] The catalog: a session scoped namespace of registered frames and views, no persistence, no schemas, no ATTACH
- [x] Bind contexts, scopes, parent pointers, correlated reference recording
- [x] Star expansion including EXCLUDE, REPLACE, RENAME, COLUMNS() and struct unpacking
- [x] Aggregate and window classification over the select list, HAVING, QUALIFY and ORDER BY
- [x] Subquery binding in four shapes, each tagged scalar, EXISTS, IN or quantified, and each marked correlated or not
- [x] CTEs including RECURSIVE, with the inline by default rule and the MATERIALIZED hints
- [x] The type lattice, implicit casts, DECIMAL128 and INT128
- [x] The tier one function registry, roughly 150 names, built at comptime
- [x] Overload resolution with DuckDB's cast cost scoring
- [x] The semantics differential harness

### The semantics that will bite

All of these were measured against DuckDB 1.5.5 rather than read from documentation, and every one of them is a silently different answer rather than an error.

`1/2` is `0.5` and a DOUBLE, because `/` on two integers is floating point division. Integer division is `//`. `1/0` is `inf` and not an error, because `ieee_floating_point_ops` defaults to true, while `1//0` is NULL. Postgres raises on both and pandas does a third thing.

`1.1 + 2.2` is `DECIMAL(3,1)` and exactly `3.3`, because an unsuffixed decimal literal is DECIMAL and not DOUBLE. `DECIMAL(4,2) * DECIMAL(5,3)` is `DECIMAL(9,5)`. `CASE WHEN true THEN 1 ELSE 2.5 END` is `DECIMAL(11,1)`, as is `coalesce(1, 2.0)` and the union of an integer and a decimal literal. So fixed point arithmetic with derived precision is not a feature to defer. Without it, arithmetic over literals gives a different answer everywhere and a large part of the corpus fails on values that look correct.

`sum()` over any integer column is HUGEINT, so INT128 is a hard requirement and not a nicety. `sum(DECIMAL(5,2))` is `DECIMAL(38,2)`, scale preserved and precision maxed. `avg(INTEGER)` is DOUBLE, `min(INTEGER)` is INTEGER, `count(*)` and `count(DISTINCT x)` are BIGINT. A sum returning BIGINT is a compatibility failure even when every value fits, because `typeof()` is observable and the corpus checks it.

`127::TINYINT + 1` raises rather than wrapping, at every width, for every arithmetic operation. So every arithmetic kernel needs a checked path, and the check has to be vectorized with a single branch on an aggregated flag rather than a branch per element. Casts raise too, and `'abc'::INTEGER` is a conversion error naming the value, with `TRY_CAST` as the escape hatch.

`'a' || NULL` is NULL and `concat('a', NULL)` is `'a'`, which is the same operation with two different null rules. `3 IN (1,2,NULL)` is NULL and so is `3 NOT IN (1,2,NULL)`, which means a NOT IN over a nullable subquery returns nothing. That is correct SQL, it surprises everyone, and it is load bearing in TPC-H q16 and q21, so the plan must never rewrite NOT IN to a plain anti join.

`default_null_order` is `NULLS_LAST` and it is absolute rather than relative to direction, so nulls sort last under both ASC and DESC. Postgres puts them first on DESC. Strings compare bytewise, so `'A' < 'a'`, and `'ab' = 'ab '` is false. `upper('ß')` is `'ẞ'`, so case folding is Unicode aware and cannot be a byte table.

Lists and strings are one based and slices include both endpoints, so `[1,2,3][1]` is `1`, `'hello'[2:3]` is `'el'`, and `substring('hello',0,3)` is `'he'` because the range is clipped and a character is lost. Every one of those disagrees with Python, which is not a reason to change it. The moment we make lists zero based, one hundred per cent compatible becomes false and the number stops meaning anything.

`VARCHAR(3)` carries no semantics at all. `typeof('a'::VARCHAR(3))` is VARCHAR and `'abcd'::VARCHAR(3)` is `'abcd'`, so the length is parsed and discarded. A helpful truncation would be a wrong answer.

TIMESTAMP is microsecond and truncates rather than rounds, while pandas defaults to nanoseconds, so that boundary is a documented loss. `now()` is TIMESTAMP WITH TIME ZONE. The position for this milestone is that naive TIMESTAMP is fully supported and TIMESTAMPTZ is supported for UTC with a named refusal otherwise, because a half implemented time zone is the worst of the three options.

### The function catalog, and why the tier is 150 and not 948

Counted from `duckdb_functions()` on 1.5.5: 2,951 overloads under 948 distinct names. 1,465 scalar overloads under 617 names, and 1,177 aggregate overloads under only 88 names, which is thirteen overloads per aggregate because each is instantiated across the numeric types and most also exist in ordered, distinct and windowed forms.

That asymmetry sets the strategy. Aggregates are few, deep and expensive per name, so they are written by hand. Scalars are many, shallow and cheap, so they are generated from a declaration list over the comptime dtype machinery `firepanda/kernel/` already uses.

Three things must never be generated, because generating them produces something wrong. Anything with a derived return type, meaning decimal arithmetic and the HUGEINT promotion. Anything whose null rule is not propagating, meaning `concat`, `coalesce`, `count` and the `||` operator, each stated explicitly at its registration site. And aggregates, because they are stateful.

### The aggregate protocol

Four operations, which is DuckDB's design and the only shape that serves both the hash aggregate and the window path: init, update over a whole input vector with a selection vector, combine, and finalize.

Update taking a vector at a time is what makes the hash aggregate fast, one call per chunk per aggregate rather than one per row. Combine is what makes it parallel across morsels and what makes it spillable, so combine is mandatory at registration. An aggregate that genuinely cannot combine declares itself ordered and gets the fallback path explicitly, which is what DuckDB's `ordered_aggregate_threshold` of 262,144 is for.

State must be fixed size and trivially relocatable wherever possible, because that is what later lets a partition spill by writing bytes.

### Exit criteria

- [ ] The semantics harness passes every case above on both the value and `typeof()`, compared against libduckdb in process
- [ ] The expression fuzzer finds no type disagreement in an overnight run
- [ ] The overload resolution fuzzer agrees with DuckDB on the resolved return type for every implemented name across a typed argument matrix
- [ ] The compile budget graph shows the registry expansion, and the number is acceptable or the fallback to a value based slow path is taken

### Depends on

S2.

Contributor guide

Open the contributing guide

Research direction

Start by reading the existing code under firepanda/kernel/ and the S2 implementation, since this issue depends on S2. Run the semantics harness against libduckdb in process and inspect the expression and overload-resolution fuzzers. Done means the listed value and typeof() cases pass, fuzzers agree with DuckDB, and the compile budget is acceptable or uses the stated fallback.

Written by the indexing model from the issue text.

Assessment

Domain
backend, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
18/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.