lance-format / lance-format/lance
perf: LIMIT/OFFSET pushdown is disabled on all stable-row-id datasets
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
Scanner::get_scan_range (rust/lance/src/dataset/scanner.rs:2896) returns Ok(None) for every dataset where manifest.uses_stable_row_ids() is true, which disables LIMIT/OFFSET pushdown entirely:
} else if self.dataset.manifest.uses_stable_row_ids() {
// Stable-row-id datasets can contain deleted / rewritten rows that still occupy
// physical positions in older fragments ...
Ok(None)
}
The reasoning in the comment is sound — filtered-read planning trims fragments before the deletion-aware remap finishes, so a pushed-down range can be spent on tombstoned positions and skip live rows in later fragments. But the guard is unconditional, so it also fires on datasets where the hazard cannot occur: a stable-row-id dataset with no deletions and no rewritten rows has physical positions that correspond exactly to visible rows.
The consequence is that a plain SELECT * FROM t LIMIT 10 on a stable-row-id dataset reads every fragment instead of stopping after the first, and the gap widens with dataset size. This is one of the cases where turning stable row ids on makes an unrelated query materially slower, which matters for the default-on decision.
A narrower guard should be possible — for example, allow pushdown when no fragment in the scan has a deletion vector and the row id sequences are contiguous ranges, which is the common append-only case. Failing that, the deletion-aware remap could run before fragment trimming.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in rust/lance/src/dataset/scanner.rs:2896 at Scanner::get_scan_range and trace how stable row IDs, deletion vectors, and fragment trimming interact. Reproduce the LIMIT/OFFSET behavior on a stable-row-id dataset without deletions or rewrites, then verify that pushdown remains safe and the scan no longer reads every fragment.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100