lance-format / lance-format/lance
Allow filtering of projected fields
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
Users can introduce new fields using projection. However, they cannot turn around and filter on these fields. This can perhaps wait until we have general SQL support but it could be implemented earlier.
The challenge is that we compile filters against the scan input schema and this does not contain the projected fields. We should compile the filter against the final projected schema. Then we can subtract fields that don't exist in the scan input schema and add a final FilterExec after the project.
This is a little bit tricky because, if we do the above directly, then the take step will try and take these fields and not find them. So we need to change the take logic slightly as well.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing filter compilation against the scan input schema, the final projected schema, and the take logic. Verify that projected fields can be filtered, fields absent from the scan input are handled appropriately, a final FilterExec follows the project, and the take step still succeeds.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100