lance-format / lance-format/lance

feature: share multi-column ZoneMap storage for column statistics

Open
#9,044 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

feature
Dominant language
Rust
Stars
7.1k
Forks
852
Avg merge
3d 18h
Merged PRs (30d)
272

Description

Parent: #8461

Goal

Support shared multi-column ZoneMap storage for column statistics. Each column must remain independently readable, with typed statistics and explicit coverage, while reducing the file and request overhead of separate indices for every column.

Existing foundation

ZoneMap segments and segment merging already exist through #6593 and #7128. #8017 added broad type support, including null statistics for nested types. Ordered versus null-only format semantics are being addressed in #8190.

Scope

  • Define a shared, column-projectable layout for multiple columns, preserving each field's type and supported statistics.
  • Support selecting a subset of columns without materializing every column's statistics.
  • Represent coverage per column, including different coverage after schema evolution or partial backfill.
  • Integrate creation, loading, updates, segment merging, and relevant rewrite/remap operations with the existing index framework.
  • Define field identity and type compatibility across rename, add, drop, and type changes.

This issue owns the storage representation and its lifecycle operations. Automatic collection policy belongs to #4522; maintenance scheduling, statistics resolution, and public API semantics are separate children of #8461. The layout does not introduce composite-key semantics or require a new persisted fragment-statistics format.

Acceptance criteria

  • One logical ZoneMap index can store statistics for multiple columns in shared files.
  • Requested columns retain typed values and can be read independently.
  • Coverage is accurate for each column across multiple fragments and segments.
  • Updates and merges preserve statistics for unaffected columns without duplicating covered data.
  • Schema evolution cannot reinterpret statistics under an incompatible field or type.
  • Tests cover heterogeneous types, null-only columns, partial coverage, and relevant rewrite/remap behavior.
  • Document the format/version contract and follow the applicable format-change process.
  • Measure projected-read I/O, file count, and memory to inform the layout and segment-size choices.

Open decisions

Physical layout, segment sizing, and caching should be chosen using the column-statistics scalability work under #8461. Shared storage should not require rewriting all existing statistics on every append.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the existing ZoneMap segment and segment-merging work referenced in #6593 and #7128, then review the scalability guidance in parent #8461. Define and validate the shared, column-projectable layout and its lifecycle operations, including independent reads, per-column coverage, schema evolution, and rewrite/remap behavior. Done requires the listed heterogeneous-type, null-only, partial-coverage, and format-contract tests plus projected-read, file-count, and memory measurements.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
database
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.