lance-format / lance-format/lance
perf: benchmark column-statistics collection, maintenance, and retrieval at scale
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
Parent: #8461
Goal
Establish reproducible correctness and resource measurements for collecting, storing, maintaining, and retrieving column statistics. Use the results to select defaults, zone size, shared storage layout, maintenance thresholds, and caching behavior.
Workloads
- Vary row count, fragment count, column count, zone size, and index-segment count independently.
- Include ordered and unordered values, narrow and wide columns, null-only/nested types, and applicable NaN values.
- Compare initial writes, appends before consolidation, fully consolidated statistics, mixed seed coverage, and backfill without seeds.
- Include deletions, updates/overlays, rewrites, and schema changes to validate coverage and precision as well as performance.
- Measure cold and warm retrieval on local and object storage, for one requested column and multiple requested columns.
Measurements
- Write throughput/latency overhead and memory from statistics collection.
- Seed and index size, file/object count, and storage overhead.
- Consolidation/backfill throughput, source scans, object requests/bytes, and peak memory.
- Statistics retrieval latency, object requests/bytes, and peak memory as rows, fragments, columns, and segments increase.
- Correctness of typed values, completeness, and precision against a small exact reference dataset.
Acceptance criteria
- Provide a reproducible benchmark harness with documented data shape, configuration, and environment.
- Separate collection, training, writing, consolidation, and retrieval costs while also measuring complete operations.
- Verify that available seeds avoid unnecessary source-data scans and that projected reads avoid unrelated columns.
- Measure null-statistics/bitmap overhead when evaluating default collection for most columns.
- Publish measured scaling limits and resource costs; distinguish actual measurements from extrapolations toward trillion-row/million-fragment tables.
- Recommend defaults and identify bounded follow-up tasks from the results.
Related work and sequencing
Start this work early, using existing implementations as a baseline. It should inform the other children of #8461 rather than wait for all of them to finish.
Cross-link #7968 for the large-table scale target. Reuse #8468/#8469 for parallel computation, and evaluate the retrieval improvements in #6778 and #6906 as they land. Training microbenchmarks alone do not establish end-to-end statistics scalability. Downstream query planning and execution benchmarks are outside this issue.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the existing column-statistics implementations and the benchmark work referenced by #8468 and #8469, then review the scale target in #7968 and retrieval work in #6778 and #6906. Build a reproducible harness that separates collection, training, writing, consolidation, and retrieval costs across the listed workloads. Done means documented measurements, correctness checks, scaling limits, and recommendations for defaults and bounded follow-up tasks.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100