Remove `Wildcard` from `Expr`
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
`Expr::Wildcard` and `Expr::QualifiedWildcard` are expressions that reference all columns. But it seems redundant that we don't need a special expr type to do that. This issue proposes to remove these two expr kinds.
### Describe the solution you'd like
Wildcard (`*`) can be expanded to concrete column lists when it appears. This manner seems viable in the three most common use cases I can come up with:
- (1) generate plan from SQL AST. Some scenarios already acted like this. E.g., `select * from table` will generate a projection with all fields, rather than a wildcard expr.
- (2) from `LogicalPlanBuilder` and (3) from `DataFrame`. In some aspects, these two entrances are the same. And both are strong typed (or strong schema-ed), which allows us to expand the wildcard immediately using the schema from current stage.
Besides this, `Expr::Wildcard` is not properly handled in the codebase, because it's not a "first class" expr. Take some functions as examples:
- `expr_to_columns`: from the correctness aspect, it should also count columns referenced by `Expr::Wildcard` and `Expr::QualifiedWildcard`.
- `create_physical_name`: this function requires all the wildcards to be expanded before calling it.
And I find another issue that related to `Wildcard`: https://github.com/apache/arrow-datafusion/issues/5473. The [solution](https://github.com/apache/arrow-datafusion/pull/5627) is to add an optimizer rule that expands all wildcards. This proposal is going to do something similar but in a more eager way.
### Describe alternatives you've considered
_No response_
### Additional context
Since `Expr` is widely used, we may need several versions to deprecate it (depending on the compatibility rule). We may ship this change step by step before fully removing these two variants
- change all the internal usage of wildcards (most of them are in aggr expr I suppose)
- change the public interface that may generate wildcard. Like https://github.com/apache/arrow-datafusion/issues/5473
- mark `Expr::Wildcard` and `Expr::QualifiedWildcard` as `#[deprecated]`
- fully remove them
Contributor guide
Research direction
Start by tracing wildcard handling through the SQL AST, LogicalPlanBuilder, and DataFrame entry points. Then inspect expr_to_columns, create_physical_name, and aggregate-expression usage, along with issue 5473 and PR 5627. Done means wildcard expressions are expanded at the supported entry points and the two Expr variants are deprecated or removed after internal uses are updated.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, sql
- Domain
- databases
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100