lance-format / lance-format/lance

feature: resolve column statistics across index and seed coverage

Open
#9,046 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

feature
Dominant language
Rust
Stars
7.1k
Forks
852
Avg merge
3d 18h
Merged PRs (30d)
272

Description

Parent: #8461

Goal

Resolve column statistics for a dataset snapshot from valid ZoneMap index coverage and compatible write-time seeds, with explicit unknown results wherever statistics are unavailable. Newly written data should be represented without waiting for central-index consolidation.

Existing foundation

#7463 provides conservative dataset-level min/max from fully covered ZoneMap indices, and #7924 handles bounds invalidated by overlays. #7427 supplies write-time seeds for index maintenance. A shared statistics resolver is still needed to combine these sources for metadata retrieval.

Resolution contract

For each requested column/statistic and relevant fragment:

  1. Use valid index statistics for the requested snapshot.
  2. Fill uncovered portions from compatible seeds, without double counting index-covered data.
  3. Return explicit unknown or incomplete coverage where neither source is usable.

Coverage and precision are separate. Full fragment coverage does not imply exact live-value statistics: deletions may leave conservative extrema and counts that describe earlier physical values. Updates or overlays may invalidate bounds. All-null data, unsupported statistics, missing statistics, and stale statistics must remain distinguishable.

Acceptance criteria

  • Resolve mixed index/seed coverage per column and statistic against the requested dataset snapshot.
  • Validate field identity, type, seed format, and relevant configuration before using a seed.
  • Avoid duplicate contributions when index segments and seeds overlap.
  • Handle append, deletion, overlays/updates, rewrites, and schema evolution without reporting invalid statistics as current.
  • Expose completeness and precision independently; never present partial aggregates as complete dataset statistics.
  • Return unsupported or unavailable statistics explicitly, including null-only and NaN cases.
  • Retrieval does not implicitly scan column values, rebuild indices, or mutate the dataset.
  • Tests cover multiple fragments and segments, mixed per-column coverage, all-null/NaN values, and mutations that introduce values outside old bounds.

Boundaries

The public typed API is tracked in #4163. Automatic seed collection belongs to #4522, and consolidation/backfill is another child of #8461. This issue supplies statistics resolution; query planning and execution changes are outside its scope. Derived or cached summaries can be evaluated without requiring a new persisted fragment-statistics format.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the foundation issues #7463, #7924, and #7427, then compare their coverage, invalidation, and seed behavior with the resolution contract here. Done means mixed index/seed statistics resolve per column and snapshot without duplicate contributions, while completeness, precision, unsupported cases, and mutations remain explicit. The linked boundaries identify #4163, #4522, and #8461 as related but out of scope.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.