lance-format / lance-format/lance
feature: support _rowoffset in scan filters beyond the IN-list take lowering
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
_rowoffset can be used in a scan filter only when the predicate lowers into TakeOperation::RowOffsets, that is a positive IN list of literal offsets. Anything else is rejected. On main at ebba5814c:
ds.scan().filter("_rowoffset > 2")
fails with Column _rowoffset does not exist from Projection::union_column, which has arms for _rowid, _rowaddr, _row_last_updated_at_version and _row_created_at_version but none for _rowoffset (rust/lance-core/src/datatypes/schema.rs:1272). Adding that arm is not sufficient on its own: the filtered read then rejects the predicate at rust/lance-datafusion/src/planner.rs:1089 with FieldNotFound { field: _rowoffset, valid_fields: [idx, _rowid, _rowaddr] }, because the column is never materialized into the schema the filter is planned against.
_rowoffset is a pure function of _rowaddr (the low 32 bits), so the shape of the fix is to request _rowaddr whenever a filter mentions _rowoffset and compute the derived column before filter refinement. That makes every predicate over it work, not just the IN shape.
This came up while fixing #9080: a negated _rowoffset list used to lower into a take of the listed offsets, which returned the complement of what was asked. That is now rejected instead, so the wrong-results bug is gone, but the predicate still cannot be answered.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with Projection::union_column in rust/lance-core/src/datatypes/schema.rs:1272 and the filtered-read planning path at rust/lance-datafusion/src/planner.rs:1089. Trace how scan filters request and materialize columns, then verify that predicates using _rowoffset beyond positive literal IN lists are accepted and produce the expected filtered rows.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 72/100