apache / apache/sedona-db

Remove vendored PushDownLeafProjections workaround after DataFusion upgrade

Open
#1,232 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
503
Forks
61
Avg merge
2d 4h
Merged PRs (30d)
90

Description

## Problem

SedonaDB currently depends on DataFusion 54.1.0, which contains a logical optimizer correctness bug in `PushDownLeafProjections` when nested-field access is applied to a struct produced by `Unnest`.

For a plan shaped like:

```text
Projection: get_field(dump, "geom")
Unnest: dump
Projection: ST_Dump(geometry) AS dump
```

the optimizer can push `get_field(dump, "geom")` below `Unnest`. At that point `dump` is still `List>`, rather than the post-unnest `Struct<...>`, and optimization fails with:

```text
Optimizer rule 'push_down_leaf_projections' failed
caused by
Execution error: Cannot access field at argument 1: type List(Struct(...))
is not Struct, Map, or Null
```

This is a plan-correctness bug, not a security issue.

## Upstream status

The bug was reported and fixed upstream:

- apache/datafusion#22615 describes `PushDownLeafProjections` crossing `Unnest`, including the same-named pre/post-unnest column problem.
- apache/datafusion#22620 fixes it by treating `Unnest` as a semantic barrier for leaf-projection pushdown.
- apache/datafusion#22783 subsequently makes the `LogicalPlan::Unnest` expression/rebuild contracts consistent while retaining the semantic barrier.

The reduced reproducer fails with DataFusion 54.1.0 and succeeds on DataFusion `main` at `e1942b144`.

## Temporary SedonaDB workaround

Until SedonaDB upgrades to a DataFusion release containing apache/datafusion#22620, `sedona-query-planner` replaces DataFusion's optimizer rule named `push_down_leaf_projections` with a compatibility rule that prevents leaf projection pushdown across `Unnest`.

Regression coverage includes both a DataFusion-only `List` plan and SedonaDB's `ST_Dump → unnest → geom` workflow.

## Action item

- [ ] After upgrading DataFusion to a release containing apache/datafusion#22620, remove the vendored `PushDownLeafProjections` compatibility rule and its registration, while retaining appropriate regression coverage against the upstream rule.

Contributor guide

Open the contributing guide

Research direction

Start by checking the DataFusion dependency version and locating the vendored compatibility rule and its registration in sedona-query-planner. Confirm the dependency includes apache/datafusion#22620, then remove the workaround while retaining the DataFusion-only List and ST_Dump → unnest → geom regression coverage. Done means the upstream rule handles both cases successfully.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases
Issue type
Refactor
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.