apache / apache/datafusion

Don't error on unknown column when pruning if predicate can still be proven false

Open
#7,869 13 comments 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

At query time, our use case requires that we evaluate predicates against in-memory data that may have a schema that is a subset of the table schema. The predicate can reference columns that are not currently in memory or known at query time.

For example, given the following in-memory data:

| col_a | value |
|--|--|
| A | 42 |

We may have to evaluate a predicate such as `col_a != A AND col_b=bananas`. Where `col_b` is not present in the in-memory schema / unknown at pruning time, but is a valid column for the table in the system as a whole.

Because at query time we have a limited subset of the schema, the schema and statistics provided when constructing the `PruningPredicate` covers only `col_a, value`.

However the `col_a != A` portion of the predicate can be proven FALSE irrespective of `col_b`. Unfortunately constructing the `PruningPredicate` eagerly validates the presence of statistics for all columns in the predicate, and errors stating that there are no fields named `col_b` before attempting to evaluate any portion of the predicate.

### Describe the solution you'd like

Attempt to evaluate the predicate based on the available statistics, and return FALSE if possible. If the predicate cannot be proven FALSE, return a "missing column" error as it does today.

For the example above, ideally pruning should return FALSE as it can be proven that `col_a != A` is FALSE even though `col_b` is unknown at pruning time.

### Describe alternatives you've considered

Inserting NULL statistics into the pruning schema to satisfy the presence check - this works around the issue, but unfortunately requires extra processing to prevent the missing field error.

### Additional context

This change in behaviour might need sticking behind a flag/option to opt into, rather than being the default.

Contributor guide

Open the contributing guide

Research direction

Start by locating the PruningPredicate construction and its eager validation of predicate columns. Reproduce the col_a/col_b example, then trace how partial predicate evaluation handles unknown columns; done means a provably false predicate returns FALSE while an unresolved predicate still reports the missing-column error.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.