apache / apache/datafusion

Allow for bounds analysis on selective columns in a schema

Open
#14,309 2 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

When performing bounds analysis on a filter expression (`Expr`), I would like to only analyze bounds for specific columns in my schema, and ignore other columns that are in the `Expr`. However, the [`analyze`](https://github.com/apache/datafusion/blob/7c07948358eac81c4b297fa2400cba3c9ca55dc2/datafusion/physical-expr/src/analysis.rs#L159) and underlying [`ExprIntervalGraph`](https://github.com/apache/datafusion/blob/7c07948358eac81c4b297fa2400cba3c9ca55dc2/datafusion/physical-expr/src/intervals/cp_solver.rs#L123) analyzes bounds for all nodes in the `Expr` tree.

The problem is that sometimes the `Expr` tree can have a mismatch in the LHS/RHS data type, which leads to the analysis failing due to errors like [this one](https://github.com/apache/datafusion/blob/7c07948358eac81c4b297fa2400cba3c9ca55dc2/datafusion/expr-common/src/interval_arithmetic.rs#L1195-L1199).

### Describe the solution you'd like

When analyzing an expression like
```
WHERE a < 10 AND b > "aaa"
```
I would like there to be a way to only analyze the bounds imposed on `a`, which in the above should result in the interval [`None`, `9`].

### Describe alternatives you've considered

I may not be using the correct APIs, or am using the correct APIs incorrectly; if either are the case, then perhaps I can learn what is correct and create an example for my use case.

There may be other solutions to this problem that I am not aware of. DataFusion may also be doing expression simplification with respect to the schema that make this less of a problem than it appears to be in my contrived [reproducer](https://github.com/apache/datafusion/pull/14308).

### Additional context

I opened a reproducer for this here: https://github.com/apache/datafusion/pull/14308

Contributor guide

Open the contributing guide

Research direction

Start with analyze in datafusion/physical-expr/src/analysis.rs and ExprIntervalGraph in datafusion/physical-expr/src/intervals/cp_solver.rs. Review reproducer PR 14308 and the interval arithmetic error location in datafusion/expr-common/src/interval_arithmetic.rs; done should include selective analysis of a while expression that returns [None, 9] for a and ignores the incompatible b condition.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.