S8. Memory, spilling and ADBC

Part of #304. Depends on S7.

Low memory is the half of the brief that is easiest to agree with and hardest to hold, because memory is not a feature you add. It is a property that every operator either preserves or destroys, and one materializing operator undoes the discipline of ten streaming ones.

This issue is last, and that is a risk. The mitigation is that the memory suite runs from S4 with published numbers, so the curve is visible long before the work to fix it starts and nobody should be surprised here.

### Scope

- [ ] Backpressure, so the number of live chunks in a pipeline is bounded structurally
- [ ] Extending the folding aggregate set, which is directly extending the set of queries that run in bounded memory
- [ ] TopN instead of sort then limit
- [ ] Radix partitioned hash aggregate and hash join, partitioned for spilling and not only for parallelism
- [ ] A memory manager with operator reservations
- [ ] Spilling for the hash aggregate and the hash join
- [ ] External sort with runs and a k way merge
- [ ] Per statement arenas for the front end, and a chunk buffer pool
- [ ] The ADBC driver
- [ ] EXPLAIN ANALYZE reporting spilled bytes per operator

### Where the memory goes

Six places and only three are large. Pipeline intermediates, which at 128K rows times columns times workers is around 100 MB at ten columns and ten workers, and is bounded by construction. Hash tables, which is the largest single consumer on TPC-H and is data dependent, since a high cardinality group by holds one state per group and there is no bound on the number of groups. Sort buffers, which are the whole input by definition unless the sort is external or is a TopN.

The other three are the input frames, which are ours and already in memory, the result, which lives as long as the user holds it, and the front end, which is kilobytes and belongs to the latency path rather than this one.

### Bounded by construction beats bounded by a check

Backpressure first. Polars bounds live morsels with a token taken at the source and returned at the sink, so chunks in flight are the token count regardless of how fast the source can produce. Our morsel queue already has the shape, since a worker pulls one morsel at a time, and the addition is that a sink which cannot accept must stop the pull rather than buffer. Without it, a slow sink behind a fast scan buffers the whole input, which is the classic streaming engine memory bug.

Then the folding aggregates. `firepanda/exec/node.mojo` already makes the distinction, holding one row per group and merging each chunk as it passes rather than holding every row until the last one, and it already says which reductions qualify is a list rather than a judgement. A sum of sums is a sum and a median of medians is not. Extending that list is the highest value memory work there is.

Then TopN. `ORDER BY x LIMIT 10` over a billion rows should hold ten rows and not a billion, which converts the most common unbounded operator into a bounded one.

### Spilling policy belongs to the operator

An allocator that fails at a limit produces an out of memory error at an arbitrary point with no way to recover. An operator that knows it is a hash aggregate over a partitioned table can decide to write partition seven to disk and carry on. DuckDB moved from the first to the second and it is the difference between a limit that raises and a limit that degrades.

Radix partitioning is what makes it possible at all. A hash aggregate or hash join partitioned by the high bits of the key hash can evict whole partitions independently and process them one at a time on rebuild, which is why the partitioning is a design requirement rather than only a parallelism device.

Fixed size relocatable aggregate state is what makes it cheap, because writing a partition is then writing bytes. An aggregate whose state is a heap allocated list, meaning `list`, `string_agg` and exact `quantile`, cannot spill and says so at registration. Those get the honest answer, which is that they can raise, and the error names the aggregate and suggests the approximate variant.

### ADBC is cheap and worth doing here

ADBC 1.1.0 is a C API over the Arrow C Data Interface, which firepanda already exports. A driver implementing database, connection and statement, with query execution returning an array stream over the result chunks. No serialization, because the stream is the chunks. That gets firepanda into anything that speaks ADBC for a few hundred lines and no new engine surface.

It is also the door where the query text came from another process, which is why it takes the restrictive capability default from S7.

### The failure mode to design against

Not running out of memory. Running out of memory at ninety five per cent of a five minute query, which is the experience that makes users leave.

Three mitigations in order of value. Reserve early, so an operator that knows it will need a large table asks before it starts and the failure lands at second one rather than minute five. Degrade rather than fail, which is what spilling is for. And report honestly, because a user whose query took thirty times longer deserves to be told why rather than left to guess.

### Exit criteria

- [ ] Peak RSS within 1.2x of DuckDB on every TPC-H query at SF10, reported per query beside the timing
- [ ] No query in the suite fails for memory that DuckDB completes
- [ ] A query at twice available memory completes, slower rather than fatally
- [ ] The group by cardinality curve from ten to a hundred million groups is published, showing the aggregate folding rather than holding
- [ ] The small statement loop allocates its chunk buffers and its front end arena once
- [ ] A third party ADBC client can connect and query

### Depends on

S7.

Read firepanda/exec/node.mojo and the memory suite from S4 first; the issue spans backpressure, folding aggregates, TopN, spilling, arenas, and ADBC rather than naming one starter task. Done means meeting the listed TPC-H memory, large-group, allocation, and third-party ADBC exit criteria.

Written by the indexing model from the issue text.