lance-format / lance-format/lance

perf: LIMIT/OFFSET pushdown is disabled on all stable-row-id datasets

Open
#8,850 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

performance
Dominant language
Rust
Stars
7.1k
Forks
852
Avg merge
3d 18h
Merged PRs (30d)
272

Description

Scanner::get_scan_range (rust/lance/src/dataset/scanner.rs:2896) returns Ok(None) for every dataset where manifest.uses_stable_row_ids() is true, which disables LIMIT/OFFSET pushdown entirely:

} else if self.dataset.manifest.uses_stable_row_ids() {
    // Stable-row-id datasets can contain deleted / rewritten rows that still occupy
    // physical positions in older fragments ...
    Ok(None)
}

The reasoning in the comment is sound — filtered-read planning trims fragments before the deletion-aware remap finishes, so a pushed-down range can be spent on tombstoned positions and skip live rows in later fragments. But the guard is unconditional, so it also fires on datasets where the hazard cannot occur: a stable-row-id dataset with no deletions and no rewritten rows has physical positions that correspond exactly to visible rows.

The consequence is that a plain SELECT * FROM t LIMIT 10 on a stable-row-id dataset reads every fragment instead of stopping after the first, and the gap widens with dataset size. This is one of the cases where turning stable row ids on makes an unrelated query materially slower, which matters for the default-on decision.

A narrower guard should be possible — for example, allow pushdown when no fragment in the scan has a deletion vector and the row id sequences are contiguous ranges, which is the common append-only case. Failing that, the deletion-aware remap could run before fragment trimming.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in rust/lance/src/dataset/scanner.rs:2896 at Scanner::get_scan_range and trace how stable row IDs, deletion vectors, and fragment trimming interact. Reproduce the LIMIT/OFFSET behavior on a stable-row-id dataset without deletions or rewrites, then verify that pushdown remains safe and the scan no longer reads every fragment.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.