apache / apache/datafusion

Don't error on unknown column when pruning if predicate can still be proven false

Aperta
#7,869 13 commenti 1 reazione 0 assegnatari Vedi su GitHub
enhancement
Lingua principale
Rust
Stelle
9.3k
Fork
2.4k
Merge medio
3g 11h
PR unite (30g)
360

Descrizione

### Is your feature request related to a problem or challenge?

At query time, our use case requires that we evaluate predicates against in-memory data that may have a schema that is a subset of the table schema. The predicate can reference columns that are not currently in memory or known at query time.

For example, given the following in-memory data:

| col_a | value |
|--|--|
| A | 42 |

We may have to evaluate a predicate such as `col_a != A AND col_b=bananas`. Where `col_b` is not present in the in-memory schema / unknown at pruning time, but is a valid column for the table in the system as a whole.

Because at query time we have a limited subset of the schema, the schema and statistics provided when constructing the `PruningPredicate` covers only `col_a, value`.

However the `col_a != A` portion of the predicate can be proven FALSE irrespective of `col_b`. Unfortunately constructing the `PruningPredicate` eagerly validates the presence of statistics for all columns in the predicate, and errors stating that there are no fields named `col_b` before attempting to evaluate any portion of the predicate.

### Describe the solution you'd like

Attempt to evaluate the predicate based on the available statistics, and return FALSE if possible. If the predicate cannot be proven FALSE, return a "missing column" error as it does today.

For the example above, ideally pruning should return FALSE as it can be proven that `col_a != A` is FALSE even though `col_b` is unknown at pruning time.

### Describe alternatives you've considered

Inserting NULL statistics into the pruning schema to satisfy the presence check - this works around the issue, but unfortunately requires extra processing to prevent the missing field error.

### Additional context

This change in behaviour might need sticking behind a flag/option to opt into, rather than being the default.

Guida per i contributori

Apri la guida per i contributori

Direzione di ricerca

Inizia individuando la costruzione di PruningPredicate e la sua validazione eager delle colonne del predicato. Riproduci l’esempio col_a/col_b, quindi traccia il modo in cui la valutazione parziale del predicato gestisce le colonne sconosciute; il lavoro è completo quando un predicato dimostrabilmente falso restituisce FALSE, mentre un predicato non risolto continua a segnalare l’errore di colonna mancante.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
rust
Ambito
databases
Tipo di issue
Funzionalità
Difficoltà
4/5
Tempo stimato
3-5 giorni
Stato di attività
Ferma
Chiarezza
Abbastanza chiara
Idoneità per principianti
42/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.