lance-format / lance-format/lance

perf: benchmark column-statistics collection, maintenance, and retrieval at scale

Open
#9,047 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

performance
Dominant language
Rust
Stars
7.1k
Forks
852
Avg merge
3d 18h
Merged PRs (30d)
272

Description

Parent: #8461

Goal

Establish reproducible correctness and resource measurements for collecting, storing, maintaining, and retrieving column statistics. Use the results to select defaults, zone size, shared storage layout, maintenance thresholds, and caching behavior.

Workloads

  • Vary row count, fragment count, column count, zone size, and index-segment count independently.
  • Include ordered and unordered values, narrow and wide columns, null-only/nested types, and applicable NaN values.
  • Compare initial writes, appends before consolidation, fully consolidated statistics, mixed seed coverage, and backfill without seeds.
  • Include deletions, updates/overlays, rewrites, and schema changes to validate coverage and precision as well as performance.
  • Measure cold and warm retrieval on local and object storage, for one requested column and multiple requested columns.

Measurements

  • Write throughput/latency overhead and memory from statistics collection.
  • Seed and index size, file/object count, and storage overhead.
  • Consolidation/backfill throughput, source scans, object requests/bytes, and peak memory.
  • Statistics retrieval latency, object requests/bytes, and peak memory as rows, fragments, columns, and segments increase.
  • Correctness of typed values, completeness, and precision against a small exact reference dataset.

Acceptance criteria

  • Provide a reproducible benchmark harness with documented data shape, configuration, and environment.
  • Separate collection, training, writing, consolidation, and retrieval costs while also measuring complete operations.
  • Verify that available seeds avoid unnecessary source-data scans and that projected reads avoid unrelated columns.
  • Measure null-statistics/bitmap overhead when evaluating default collection for most columns.
  • Publish measured scaling limits and resource costs; distinguish actual measurements from extrapolations toward trillion-row/million-fragment tables.
  • Recommend defaults and identify bounded follow-up tasks from the results.

Related work and sequencing

Start this work early, using existing implementations as a baseline. It should inform the other children of #8461 rather than wait for all of them to finish.

Cross-link #7968 for the large-table scale target. Reuse #8468/#8469 for parallel computation, and evaluate the retrieval improvements in #6778 and #6906 as they land. Training microbenchmarks alone do not establish end-to-end statistics scalability. Downstream query planning and execution benchmarks are outside this issue.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the existing column-statistics implementations and the benchmark work referenced by #8468 and #8469, then review the scale target in #7968 and retrieval work in #6778 and #6906. Build a reproducible harness that separates collection, training, writing, consolidation, and retrieval costs across the listed workloads. Done means documented measurements, correctness checks, scaling limits, and recommendations for defaults and bounded follow-up tasks.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.