apache / apache/spark

PushPredicateThroughNonJoin assertion failure when pushing filter through Project into Union (Hive view)

Closed
#55,575 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Scala
Stars
44k
Forks
29.4k
PR merge metrics
No merged PRs in 30d

Description

### Describe the bug

`PushPredicateThroughNonJoin` crashes with `AssertionError` when a filter references a passthrough column (not aliased) on a CTE/subquery that reads from a Hive view backed by a UNION ALL.

### Reproduction

Given a Hive view `my_view` defined as `SELECT * FROM t1 UNION ALL SELECT * FROM t2 UNION ALL SELECT * FROM t3`:

```sql
-- Crashes
WITH cte AS (
SELECT id, name, col1['key'] AS col1_alias
FROM my_view
WHERE ds = '2024-01-01'
)
SELECT * FROM cte WHERE name = 'foo';
```

```sql
-- Works (filter directly on the view, no CTE/Project in between)
SELECT * FROM my_view WHERE name = 'foo' AND ds = '2024-01-01';
```

```sql
-- Works (filter inside the CTE, below the Project, directly above Union)
WITH cte AS (
SELECT id, name, col1['key'] AS col1_alias
FROM my_view
WHERE ds = '2024-01-01'
AND name = 'foo'
)
SELECT * FROM cte;
```

### Root Cause

The rule applies in two iterations via `plan transform applyLocally`:

**Iteration 1 — Filter + Project case** (`Optimizer.scala` ~line 1724):

```scala
case Filter(condition, project @ Project(fields, grandChild)) =>
val aliasMap = getAliasMap(project)
project.copy(child = Filter(replaceAlias(condition, aliasMap), grandChild))
```

Pushes `name = 'foo'` below the Project. Since `name` is a passthrough column (not in `aliasMap`), `replaceAlias` returns the attribute with its **original exprId** from the Project output context (`name#X`).

Result: `Project(..., Filter(name#X = 'foo', Union))`

**Iteration 2 — Filter + Union case** (~line 1783):

```scala
case filter @ Filter(condition, union: Union) =>
val output = union.output
val newGrandChildren = union.children.map { grandchild =>
val newCond = pushDownCond transform {
case e if output.exists(_.semanticEquals(e)) =>
grandchild.output(output.indexWhere(_.semanticEquals(e)))
}
assert(newCond.references.subsetOf(grandchild.outputSet)) // FAILS
Filter(newCond, grandchild)
}
```

The Union's `output` has `name#Y` (derived from `firstAttr.exprId` in `Union.output`). `semanticEquals` compares exprIds, so `name#X` does not match `name#Y`. The attribute is never remapped, and the assertion fails.

The exprId mismatch occurs because when a Hive view is expanded, the view resolution layer assigns new exprIds to the Union's output attributes that differ from the exprIds in the Project's output references.

### Workaround

Move the filter into the CTE/subquery so it sits directly above the Union, skipping the Filter → Project → Union code path.

### Stack Trace

```
java.lang.AssertionError: assertion failed
at scala.Predef$.assert(Predef.scala:208)
at o.a.s.sql.catalyst.optimizer.PushPredicateThroughNonJoin$$anonfun$7.$anonfun$applyOrElse$55(Optimizer.scala:2040)
at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
at scala.collection.TraversableLike.map(TraversableLike.scala:286)
at o.a.s.sql.catalyst.optimizer.PushPredicateThroughNonJoin$$anonfun$7.applyOrElse(Optimizer.scala:2035)
at o.a.s.sql.catalyst.optimizer.PushPredicateThroughNonJoin$$anonfun$7.applyOrElse(Optimizer.scala:1962)
at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:175)
at o.a.s.sql.catalyst.trees.TreeNode.$anonfun$transformDownWithPruning$1(TreeNode.scala:521)
at o.a.s.sql.catalyst.optimizer.PushDownPredicates$.apply(Optimizer.scala:1948)
```

### Environment

- Spark version: 3.5.5 (AMZ fork `3.5.5-amzn-1`)
- Java version: JDK 17
- Scala version: 2.12.18
- Running on EMR with Hive metastore and Iceberg tables behind the UNION view

Contributor guide

Open the contributing guide

Research direction

Start in Optimizer.scala at PushPredicateThroughNonJoin, especially the Filter + Project and Filter + Union cases described in the issue. Reproduce the provided Hive-view CTE query and compare it with the working variants; done means the predicate no longer triggers the assertion and is correctly handled across the Union.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, sql
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.