lance-format / lance-format/lance

Use DF's planner directly for expression (filter) parsing

Open
#1,129 3 comments 0 reactions 1 assignee View on GitHub

@westonpace is already working on this.

Since Aug 9, 2023.

enhancement
Dominant language
Rust
Stars
7.1k
Forks
852
Avg merge
3d 18h
Merged PRs (30d)
272

Description

Currently, lance implements its own parser/planner logic in planner.rs in order to convert filters from SQL to datafusion expressions. This means we have to make extensions to lance's planner/parser to expose new capabilities as DF releases them (e.g. https://github.com/lancedb/lance/issues/1115 ). It would be easier to reuse DF's planner/parser directly. I've started an experiment to do this here: https://github.com/lancedb/lance/compare/main...westonpace:lance:experiment/use-df-planner?expand=1

There are a few prerequisite tasks before we can complete this.

  • https://github.com/lancedb/lance/issues/849 - Lance's planner is designed to enable pyarrow.compute.Expression -> string -> DF expression. So it makes some choices (like a default timestamp resolution of microseconds) that are different than datafusion's planner. If we can instead go pyarrow.compute.Expression -> substrait -> DF then we don't need to worry about this.
  • https://github.com/apache/arrow-datafusion/issues/7249 - Lance's planner is capable of handling timestamp literals with a specified precision (to choose between second/millisecond/microsecond/nanosecond resolution). Datafusion's planner can't yet handle this. So we need to add this capability to DF if we want to avoid a regression.
  • we need to investigate / confirm DF's capability to handle UDFs and UDTs. Example from @eddyxu : we might want “Select * From … Where image_size(img) >= (256, 512)”
  • we need to somehow support expressions from duckdb. One example of a problem here is that duckdb has a function is_valid and datafusion expects IS NOT NULL (not a function). Substrait could be a solution here if we write a converter (there are some seeds for this).
  • Datafusion does not support nested nested references. For example, x.y is supported but not x.y.z

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.