lance-format / lance-format/lance

Support max_source_rows and max_source_bytes limits

Open
#8,234 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Rust
Stars
7.1k
Forks
852
Avg merge
3d 18h
Merged PRs (30d)
272

Description

Why

On large datasets compaction has to run incrementally — no single run can afford to
rewrite everything at once. Today the only knob for that is max_source_fragments.
It works well when fragments are similarly sized, but fragment sizes can vary a lot
in practice (a fragment may hold a handful of live rows or millions), so the same
fragment budget may translate into very different amounts of IO and rewrite time
across runs. Row- and byte-based budgets make the per-run workload easier to predict.

What

Add two planning budgets to CompactionOptions that complement max_source_fragments
by bounding a single run on the data volume dimension:

  • max_source_rows — counted as live rows (physical rows minus soft-deleted rows).
  • max_source_bytes — total size of the source fragments' data and overlay files,
    read from sizes recorded in the manifest (no object-store metadata round trips).
    Deletion files and files that only back dropped columns are not counted.

Behavior:

  • All configured max_source_* limits apply together: tasks are admitted in order
    until adding the next task would exceed any one of them.
  • The limits are hard upper bounds: if the first task already exceeds a budget the
    plan is empty and a warning is logged.
  • A budget of zero is rejected (None means no limit); validation is centralized in
    CompactionOptions::validate.

Exposed in Rust / Python / Java, plus the manifest config keys
lance.compaction.max_source_rows and lance.compaction.max_source_bytes.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at CompactionOptions and its validate method, then trace compaction planning and the Rust, Python, and Java configuration surfaces. Verify that combined row, byte, and fragment limits, zero-value rejection, oversized-first-task handling, manifest config keys, and the stated byte-counting rules are implemented consistently.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, python, rust
Domain
data-engineering, databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.