apache / apache/datafusion

Remove `Wildcard` from `Expr`

Aperta
#7,765 8 commenti 0 reazioni 0 assegnatari Vedi su GitHub
enhancement
Lingua principale
Rust
Stelle
9.3k
Fork
2.4k
Merge medio
3g 11h
PR unite (30g)
360

Descrizione

### Is your feature request related to a problem or challenge?

`Expr::Wildcard` and `Expr::QualifiedWildcard` are expressions that reference all columns. But it seems redundant that we don't need a special expr type to do that. This issue proposes to remove these two expr kinds.

### Describe the solution you'd like

Wildcard (`*`) can be expanded to concrete column lists when it appears. This manner seems viable in the three most common use cases I can come up with:
- (1) generate plan from SQL AST. Some scenarios already acted like this. E.g., `select * from table` will generate a projection with all fields, rather than a wildcard expr.
- (2) from `LogicalPlanBuilder` and (3) from `DataFrame`. In some aspects, these two entrances are the same. And both are strong typed (or strong schema-ed), which allows us to expand the wildcard immediately using the schema from current stage.

Besides this, `Expr::Wildcard` is not properly handled in the codebase, because it's not a "first class" expr. Take some functions as examples:
- `expr_to_columns`: from the correctness aspect, it should also count columns referenced by `Expr::Wildcard` and `Expr::QualifiedWildcard`.
- `create_physical_name`: this function requires all the wildcards to be expanded before calling it.

And I find another issue that related to `Wildcard`: https://github.com/apache/arrow-datafusion/issues/5473. The [solution](https://github.com/apache/arrow-datafusion/pull/5627) is to add an optimizer rule that expands all wildcards. This proposal is going to do something similar but in a more eager way.

### Describe alternatives you've considered

_No response_

### Additional context

Since `Expr` is widely used, we may need several versions to deprecate it (depending on the compatibility rule). We may ship this change step by step before fully removing these two variants
- change all the internal usage of wildcards (most of them are in aggr expr I suppose)
- change the public interface that may generate wildcard. Like https://github.com/apache/arrow-datafusion/issues/5473
- mark `Expr::Wildcard` and `Expr::QualifiedWildcard` as `#[deprecated]`
- fully remove them

Guida per i contributori

Apri la guida per i contributori

Direzione di ricerca

Inizia tracciando la gestione dei wildcard attraverso il SQL AST, LogicalPlanBuilder e i punti di ingresso di DataFrame. Poi esamina expr_to_columns, create_physical_name e l’uso delle espressioni di aggregazione, insieme all’issue 5473 e alla PR 5627. Il lavoro è completato quando le espressioni wildcard vengono espanse nei punti di ingresso supportati e le due varianti di Expr sono deprecate o rimosse dopo l’aggiornamento degli utilizzi interni.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
rust, sql
Ambito
databases
Tipo di issue
Refactoring
Difficoltà
5/5
Tempo stimato
Più di una settimana
Stato di attività
Ferma
Chiarezza
Abbastanza chiara
Idoneità per principianti
35/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.