lance-format / lance-format/lance
feature: share multi-column ZoneMap storage for column statistics
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
Parent: #8461
Goal
Support shared multi-column ZoneMap storage for column statistics. Each column must remain independently readable, with typed statistics and explicit coverage, while reducing the file and request overhead of separate indices for every column.
Existing foundation
ZoneMap segments and segment merging already exist through #6593 and #7128. #8017 added broad type support, including null statistics for nested types. Ordered versus null-only format semantics are being addressed in #8190.
Scope
- Define a shared, column-projectable layout for multiple columns, preserving each field's type and supported statistics.
- Support selecting a subset of columns without materializing every column's statistics.
- Represent coverage per column, including different coverage after schema evolution or partial backfill.
- Integrate creation, loading, updates, segment merging, and relevant rewrite/remap operations with the existing index framework.
- Define field identity and type compatibility across rename, add, drop, and type changes.
This issue owns the storage representation and its lifecycle operations. Automatic collection policy belongs to #4522; maintenance scheduling, statistics resolution, and public API semantics are separate children of #8461. The layout does not introduce composite-key semantics or require a new persisted fragment-statistics format.
Acceptance criteria
- One logical ZoneMap index can store statistics for multiple columns in shared files.
- Requested columns retain typed values and can be read independently.
- Coverage is accurate for each column across multiple fragments and segments.
- Updates and merges preserve statistics for unaffected columns without duplicating covered data.
- Schema evolution cannot reinterpret statistics under an incompatible field or type.
- Tests cover heterogeneous types, null-only columns, partial coverage, and relevant rewrite/remap behavior.
- Document the format/version contract and follow the applicable format-change process.
- Measure projected-read I/O, file count, and memory to inform the layout and segment-size choices.
Open decisions
Physical layout, segment sizing, and caching should be chosen using the column-statistics scalability work under #8461. Shared storage should not require rewriting all existing statistics on every append.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the existing ZoneMap segment and segment-merging work referenced in #6593 and #7128, then review the scalability guidance in parent #8461. Define and validate the shared, column-projectable layout and its lifecycle operations, including independent reads, per-column coverage, schema evolution, and rewrite/remap behavior. Done requires the listed heterogeneous-type, null-only, partial-coverage, and format-contract tests plus projected-read, file-count, and memory measurements.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- database
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100