feature: automatic ZoneMap-backed column statistics

Goal

Provide a stable, typed, multi-column statistics API backed by ZoneMap indices and write-time seeds. Statistics for newly written data should be available without requiring compaction. Results must explicitly describe coverage and precision.

This roadmap covers column-statistics collection, storage, maintenance, retrieval, correctness, and scalability.

Background

PR #5639 proposed a separate dataset-level column-statistics system. It was closed in favor of reusing the generic index/seed infrastructure, avoiding duplicated write-time collection and a column-statistics-specific manifest mechanism. The revised direction is described in discussion #4540.

Current baseline

Status checked on September 7, 2026:

• Write-time seeds: #7427 merged. Eligible existing indices can collect seeds during append, and index updates can harvest them. Automatic collection from initial dataset creation and read-time statistics resolution across indices and seeds remain follow-up work.
• Type support: #8017 merged. Nested types receive null statistics, not ordered min/max. #8190 is open and currently addresses the null-only v1 format boundary; ordered extrema enhancements are deferred. Track capabilities through #7987 and NaN semantics through #4471.
• Statistics APIs: #7463 merged a conservative Rust min/max accessor; #6421 merged Java per-zone statistics. A general typed multi-column API with explicit coverage and precision remains to be delivered.
• Storage primitives: #6593 and #7128 merged ZoneMap segments and segment merging. Reuse these for shared storage and incremental maintenance.
• Performance work: #8469 is open and explicitly waiting for ZoneMap v1. #6778 and #6906 are open improvements to statistics retrieval through Java.

Delivery contract

• Collect statistics during writes according to a dataset/column policy, including initial writes without an existing index.
• Resolve statistics for the requested dataset snapshot from valid index coverage, then compatible seeds, then explicit unknown results.
• Maintain consolidated indices separately from write-time collection. Newly written statistics should not depend on waiting for consolidation or compaction.
• Report completeness separately from precision. Conservative bounds or physical-value counts after deletion must not be presented as exact live statistics.
• Preserve typed values and distinguish all-null, unsupported, unavailable, and stale statistics. Never report partial aggregates as complete dataset statistics.
• Keep ordinary retrieval free of implicit source-data scans, index builds, or dataset mutations. Backfill belongs to explicit maintenance.

Child issues

1. Type capabilities and format correctness

• #7987

Record the completed #8017 baseline and track remaining capabilities and #8190's format/implementation work. Define supported statistics per type and coordinate with #4471 for NaN semantics.

2. Shared multi-column ZoneMap storage

• #9044

Provide typed, column-projectable shared storage, independent per-column coverage, and lifecycle support for schema changes and segment updates/merging.

3. Automatic write-time statistics collection

• #4522

Define eligible types, dataset/column defaults, and opt-out. Collect seeds during initial writes, appends, and relevant rewrites without requiring an existing index. Account for updates and schema changes when determining validity.

4. Incremental consolidation and backfill

• #9045

Harvest compatible seeds, scan only missing data, and use existing segment APIs to create, refresh, and merge statistics. Define maintenance triggers and resource limits without requiring compaction.

5. Coverage-aware statistics resolution

• #9046

Resolve index and seed coverage for each requested column/statistic and snapshot. Handle mutations safely, avoid duplicate contributions, and report unknown results and precision explicitly.

6. Stable typed column-statistics API

• #4163

Support bulk column requests and typed min/max/null/NaN statistics where applicable, with explicit completeness and precision. Build on existing APIs and provide consistent Rust, Python, and Java surfaces. Keep richer statistics as extensions beyond the initial contract.

7. Parallel statistics computation

• #8468

Track #8469 and its ZoneMap v1 dependency. Training microbenchmarks do not replace complete-operation scalability measurements.

8. Column-statistics scalability validation

• #9047

Measure collection overhead, consolidation/backfill throughput, cold/warm retrieval latency, storage/object costs, and memory across increasing rows, fragments, columns, and segments. Include mixed seed coverage and mutations. Cross-link #7968 for the large-table scale target.

Sequencing and open decisions

Start scalability measurements early using the current implementation. Use the results to guide defaults, zone size, shared-file layout, segment size, maintenance thresholds, and caching. Storage and write-time collection can progress alongside the API contract; the resolver connects those sources to the API. #8469 currently waits for ZoneMap v1.

The remaining design decisions are the eligible/default column policy and rollout, physical layout and sizing, maintenance triggers/budgets, and the precise API representation of typed values, coverage, and precision. The requirements above do not prescribe a new persisted fragment-statistics format.