apache / apache/datafusion

Remove `Wildcard` from `Expr`

Open
#7,765 8 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Is your feature request related to a problem or challenge?

`Expr::Wildcard` and `Expr::QualifiedWildcard` are expressions that reference all columns. But it seems redundant that we don't need a special expr type to do that. This issue proposes to remove these two expr kinds.

### Describe the solution you'd like

Wildcard (`*`) can be expanded to concrete column lists when it appears. This manner seems viable in the three most common use cases I can come up with:
- (1) generate plan from SQL AST. Some scenarios already acted like this. E.g., `select * from table` will generate a projection with all fields, rather than a wildcard expr.
- (2) from `LogicalPlanBuilder` and (3) from `DataFrame`. In some aspects, these two entrances are the same. And both are strong typed (or strong schema-ed), which allows us to expand the wildcard immediately using the schema from current stage.

Besides this, `Expr::Wildcard` is not properly handled in the codebase, because it's not a "first class" expr. Take some functions as examples:
- `expr_to_columns`: from the correctness aspect, it should also count columns referenced by `Expr::Wildcard` and `Expr::QualifiedWildcard`.
- `create_physical_name`: this function requires all the wildcards to be expanded before calling it.

And I find another issue that related to `Wildcard`: https://github.com/apache/arrow-datafusion/issues/5473. The [solution](https://github.com/apache/arrow-datafusion/pull/5627) is to add an optimizer rule that expands all wildcards. This proposal is going to do something similar but in a more eager way.

### Describe alternatives you've considered

_No response_

### Additional context

Since `Expr` is widely used, we may need several versions to deprecate it (depending on the compatibility rule). We may ship this change step by step before fully removing these two variants
- change all the internal usage of wildcards (most of them are in aggr expr I suppose)
- change the public interface that may generate wildcard. Like https://github.com/apache/arrow-datafusion/issues/5473
- mark `Expr::Wildcard` and `Expr::QualifiedWildcard` as `#[deprecated]`
- fully remove them

Contributor guide

Open the contributing guide

Research direction

Start by tracing wildcard handling through the SQL AST, LogicalPlanBuilder, and DataFrame entry points. Then inspect expr_to_columns, create_physical_name, and aggregate-expression usage, along with issue 5473 and PR 5627. Done means wildcard expressions are expanded at the supported entry points and the two Expr variants are deprecated or removed after internal uses are updated.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, sql
Domain
databases
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.