lance-format / lance-format/lance
Support max_source_rows and max_source_bytes limits
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
Why
On large datasets compaction has to run incrementally — no single run can afford to
rewrite everything at once. Today the only knob for that is max_source_fragments.
It works well when fragments are similarly sized, but fragment sizes can vary a lot
in practice (a fragment may hold a handful of live rows or millions), so the same
fragment budget may translate into very different amounts of IO and rewrite time
across runs. Row- and byte-based budgets make the per-run workload easier to predict.
What
Add two planning budgets to CompactionOptions that complement max_source_fragments
by bounding a single run on the data volume dimension:
max_source_rows— counted as live rows (physical rows minus soft-deleted rows).max_source_bytes— total size of the source fragments' data and overlay files,
read from sizes recorded in the manifest (no object-store metadata round trips).
Deletion files and files that only back dropped columns are not counted.
Behavior:
- All configured
max_source_*limits apply together: tasks are admitted in order
until adding the next task would exceed any one of them. - The limits are hard upper bounds: if the first task already exceeds a budget the
plan is empty and a warning is logged. - A budget of zero is rejected (
Nonemeans no limit); validation is centralized in
CompactionOptions::validate.
Exposed in Rust / Python / Java, plus the manifest config keys
lance.compaction.max_source_rows and lance.compaction.max_source_bytes.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at CompactionOptions and its validate method, then trace compaction planning and the Rust, Python, and Java configuration surfaces. Verify that combined row, byte, and fragment limits, zero-value rejection, oversized-first-task handling, manifest config keys, and the stated byte-counting rules are implemented consistently.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, python, rust
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100