lance-format / lance-format/lance

feature: support _rowoffset in scan filters beyond the IN-list take lowering

Open
#9,105 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

feature
Dominant language
Rust
Stars
7.1k
Forks
852
Avg merge
3d 18h
Merged PRs (30d)
272

Description

_rowoffset can be used in a scan filter only when the predicate lowers into TakeOperation::RowOffsets, that is a positive IN list of literal offsets. Anything else is rejected. On main at ebba5814c:

ds.scan().filter("_rowoffset > 2")

fails with Column _rowoffset does not exist from Projection::union_column, which has arms for _rowid, _rowaddr, _row_last_updated_at_version and _row_created_at_version but none for _rowoffset (rust/lance-core/src/datatypes/schema.rs:1272). Adding that arm is not sufficient on its own: the filtered read then rejects the predicate at rust/lance-datafusion/src/planner.rs:1089 with FieldNotFound { field: _rowoffset, valid_fields: [idx, _rowid, _rowaddr] }, because the column is never materialized into the schema the filter is planned against.

_rowoffset is a pure function of _rowaddr (the low 32 bits), so the shape of the fix is to request _rowaddr whenever a filter mentions _rowoffset and compute the derived column before filter refinement. That makes every predicate over it work, not just the IN shape.

This came up while fixing #9080: a negated _rowoffset list used to lower into a take of the listed offsets, which returned the complement of what was asked. That is now rejected instead, so the wrong-results bug is gone, but the predicate still cannot be answered.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with Projection::union_column in rust/lance-core/src/datatypes/schema.rs:1272 and the filtered-read planning path at rust/lance-datafusion/src/planner.rs:1089. Trace how scan filters request and materialize columns, then verify that predicates using _rowoffset beyond positive literal IN lists are accepted and produce the expected filtered rows.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.