lance-format / lance-format/lance
Use DF's planner directly for expression (filter) parsing
@westonpace is already working on this.
Since Aug 9, 2023.
- Dominant language
- Rust
- Stars
- 7.1k
- Forks
- 852
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 272
Description
Currently, lance implements its own parser/planner logic in planner.rs in order to convert filters from SQL to datafusion expressions. This means we have to make extensions to lance's planner/parser to expose new capabilities as DF releases them (e.g. https://github.com/lancedb/lance/issues/1115 ). It would be easier to reuse DF's planner/parser directly. I've started an experiment to do this here: https://github.com/lancedb/lance/compare/main...westonpace:lance:experiment/use-df-planner?expand=1
There are a few prerequisite tasks before we can complete this.
- https://github.com/lancedb/lance/issues/849 - Lance's planner is designed to enable
pyarrow.compute.Expression -> string -> DFexpression. So it makes some choices (like a default timestamp resolution of microseconds) that are different than datafusion's planner. If we can instead gopyarrow.compute.Expression -> substrait -> DFthen we don't need to worry about this. - https://github.com/apache/arrow-datafusion/issues/7249 - Lance's planner is capable of handling timestamp literals with a specified precision (to choose between second/millisecond/microsecond/nanosecond resolution). Datafusion's planner can't yet handle this. So we need to add this capability to DF if we want to avoid a regression.
- we need to investigate / confirm DF's capability to handle UDFs and UDTs. Example from
@eddyxu:we might want “Select * From … Where image_size(img) >= (256, 512)” - we need to somehow support expressions from duckdb. One example of a problem here is that duckdb has a function
is_validand datafusion expectsIS NOT NULL(not a function). Substrait could be a solution here if we write a converter (there are some seeds for this). - Datafusion does not support nested nested references. For example,
x.yis supported but notx.y.z
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.