Remove vendored PushDownLeafProjections workaround after DataFusion upgrade
- Dominant language
- Rust
- Stars
- 503
- Forks
- 61
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 90
Description
## Problem
SedonaDB currently depends on DataFusion 54.1.0, which contains a logical optimizer correctness bug in `PushDownLeafProjections` when nested-field access is applied to a struct produced by `Unnest`.
For a plan shaped like:
```text
Projection: get_field(dump, "geom")
Unnest: dump
Projection: ST_Dump(geometry) AS dump
```
the optimizer can push `get_field(dump, "geom")` below `Unnest`. At that point `dump` is still `List>`, rather than the post-unnest `Struct<...>`, and optimization fails with:
```text
Optimizer rule 'push_down_leaf_projections' failed
caused by
Execution error: Cannot access field at argument 1: type List(Struct(...))
is not Struct, Map, or Null
```
This is a plan-correctness bug, not a security issue.
## Upstream status
The bug was reported and fixed upstream:
- apache/datafusion#22615 describes `PushDownLeafProjections` crossing `Unnest`, including the same-named pre/post-unnest column problem.
- apache/datafusion#22620 fixes it by treating `Unnest` as a semantic barrier for leaf-projection pushdown.
- apache/datafusion#22783 subsequently makes the `LogicalPlan::Unnest` expression/rebuild contracts consistent while retaining the semantic barrier.
The reduced reproducer fails with DataFusion 54.1.0 and succeeds on DataFusion `main` at `e1942b144`.
## Temporary SedonaDB workaround
Until SedonaDB upgrades to a DataFusion release containing apache/datafusion#22620, `sedona-query-planner` replaces DataFusion's optimizer rule named `push_down_leaf_projections` with a compatibility rule that prevents leaf projection pushdown across `Unnest`.
Regression coverage includes both a DataFusion-only `List` plan and SedonaDB's `ST_Dump → unnest → geom` workflow.
## Action item
- [ ] After upgrading DataFusion to a release containing apache/datafusion#22620, remove the vendored `PushDownLeafProjections` compatibility rule and its registration, while retaining appropriate regression coverage against the upstream rule.
Contributor guide
Research direction
Start by checking the DataFusion dependency version and locating the vendored compatibility rule and its registration in sedona-query-planner. Confirm the dependency includes apache/datafusion#22620, then remove the workaround while retaining the DataFusion-only List and ST_Dump → unnest → geom regression coverage. Done means the upstream rule handles both cases successfully.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- databases
- Issue type
- Refactor
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100