lance-format / lance-format/lance
feature: resolve column statistics across index and seed coverage
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
Parent: #8461
Goal
Resolve column statistics for a dataset snapshot from valid ZoneMap index coverage and compatible write-time seeds, with explicit unknown results wherever statistics are unavailable. Newly written data should be represented without waiting for central-index consolidation.
Existing foundation
#7463 provides conservative dataset-level min/max from fully covered ZoneMap indices, and #7924 handles bounds invalidated by overlays. #7427 supplies write-time seeds for index maintenance. A shared statistics resolver is still needed to combine these sources for metadata retrieval.
Resolution contract
For each requested column/statistic and relevant fragment:
- Use valid index statistics for the requested snapshot.
- Fill uncovered portions from compatible seeds, without double counting index-covered data.
- Return explicit unknown or incomplete coverage where neither source is usable.
Coverage and precision are separate. Full fragment coverage does not imply exact live-value statistics: deletions may leave conservative extrema and counts that describe earlier physical values. Updates or overlays may invalidate bounds. All-null data, unsupported statistics, missing statistics, and stale statistics must remain distinguishable.
Acceptance criteria
- Resolve mixed index/seed coverage per column and statistic against the requested dataset snapshot.
- Validate field identity, type, seed format, and relevant configuration before using a seed.
- Avoid duplicate contributions when index segments and seeds overlap.
- Handle append, deletion, overlays/updates, rewrites, and schema evolution without reporting invalid statistics as current.
- Expose completeness and precision independently; never present partial aggregates as complete dataset statistics.
- Return unsupported or unavailable statistics explicitly, including null-only and NaN cases.
- Retrieval does not implicitly scan column values, rebuild indices, or mutate the dataset.
- Tests cover multiple fragments and segments, mixed per-column coverage, all-null/NaN values, and mutations that introduce values outside old bounds.
Boundaries
The public typed API is tracked in #4163. Automatic seed collection belongs to #4522, and consolidation/backfill is another child of #8461. This issue supplies statistics resolution; query planning and execution changes are outside its scope. Derived or cached summaries can be evaluated without requiring a new persisted fragment-statistics format.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the foundation issues #7463, #7924, and #7427, then compare their coverage, invalidation, and seed behavior with the resolution contract here. Done means mixed index/seed statistics resolve per column and snapshot without duplicate contributions, while completeness, precision, unsupported cases, and mutations remain explicit. The linked boundaries identify #4163, #4522, and #8461 as related but out of scope.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100