apache / apache/datafusion

partition_by and filter on ExprFunctionExt

Open
#21,697 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

When chaining methods from ExprFunctionExt, the outcome depends on whether the first call is made on an Expr or on an ExprFuncBuilder. The impl ... for Expr has narrower per-method support than impl ... for ExprFuncBuilder, so the same set of calls in a different order can succeed or fail.
Concretely, Expr::filter only matches AggregateFunction and Expr::partition_by only matches WindowFunction. Other variants fall through to an empty builder, which then errors on .build(). The ExprFuncBuilder impl simply sets the relevant field without any matching.
The result is that this fails:
```rust
sum(col("x"))
.filter(col("a").gt_eq(lit(5)))
.partition_by(vec![col("y")])
.build()?;
```
…and depending on which clause you lead with, you can end up either with a .build() error or with a builder whose later fields were silently set. Reversing the order of .filter and .partition_by for an aggregate likewise changes behavior:
```rust
sum(col("x"))
.partition_by(vec![col("y")])
.filter(col("a").gt_eq(lit(5)))
.build()?;
```
So call order determines whether the chain errors.

Is this by design, or a genuinely incomplete part? The Expr impl of order_by and null_treatment already handles both AggregateFunction and WindowFunction; extending filter and partition_by the same way would make behavior order-independent and match what ExprFuncBuilder already does. Happy to send a PR if that's the preferred direction.

Contributor guide

Open the contributing guide

Research direction

Start at the Expr implementations of filter and partition_by, then compare their handling with order_by and null_treatment and with ExprFuncBuilder. Confirm the intended behavior for aggregate and window functions, and consider the two chaining examples as the completion criteria: both clause orders should behave consistently rather than failing during build.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.