lance-format / lance-format/lance
feature: incrementally consolidate and backfill column statistics
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
Parent: #8461
Goal
Build and maintain consolidated column statistics from write-time ZoneMap seeds, scanning only data that lacks usable statistics. Updating statistics should not require data compaction.
Existing foundation
#7427 added seed harvesting during index updates. #6593 and #7128 provide ZoneMap segments and merging. The current seed-harvesting path is all-or-nothing: missing seed data can cause the update to scan all unindexed data. This work should support mixed seed and scan inputs incrementally.
Scope
- Backfill existing fragments when statistics are first enabled.
- Harvest compatible seeds and scan only the uncovered fragments/columns that require backfill.
- Create, refresh, and merge central index segments through the existing index APIs.
- Define automatic maintenance triggers, thresholds, and bounded resource use, with an explicit invocation path for callers.
- Commit statistics together with accurate coverage for the intended dataset snapshot, respecting concurrent writes, schema changes, and index drops.
Write-time seed collection and column policy belong to #4522. Read-side coverage resolution and the public statistics API are separate children of #8461. Ordinary statistics retrieval must not silently start this maintenance work.
Acceptance criteria
- Existing datasets can be backfilled without requiring compaction.
- An update combines compatible seeds with scans of only missing data.
- Fully seeded input updates statistics without scanning source column values.
- Incremental refresh and segment merging avoid double counting and preserve per-column coverage.
- Document trigger policy, resource limits, and retry behavior.
- Concurrent append, delete, rewrite, schema changes, and index drop cannot publish incorrect coverage or restore a dropped index.
- Tests cover multiple fragments and segments, mixed seed availability, incompatible seeds, and interrupted/retried work.
- Record scan I/O, seed-read I/O, refresh throughput, and peak memory in the column-statistics benchmark.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with parent issue #8461 and the existing index APIs described in #7427, #6593, and #7128 to understand seed harvesting, ZoneMap segments, and merging. Use the column-statistics benchmark as the entry point for measuring scan I/O, seed-read I/O, throughput, and memory. Done means incremental backfill and refresh handle mixed or incompatible seeds, concurrent changes, retries, and index drops while meeting the listed acceptance tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100