apache / apache/iceberg

Spark 4.1 incompatible: Create View

Open
#15,238 16 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 11h
Merged PRs (30d)
132

Description

### Apache Iceberg version

1.10.1 (latest release)

### Query engine

Spark

### Please describe the bug 🐞

```python
CREATE OR REPLACE VIEW catalog.namespace.customer
AS SELECT *
FROM catalog.namespace.customer
```
The error:
```
Py4JJavaError: An error occurred while calling o160.sql.
: java.lang.NoSuchMethodError: 'void org.apache.spark.sql.catalyst.analysis.ResolvedIdentifier.(org.apache.spark.sql.connector.catalog.CatalogPlugin, org.apache.spark.sql.connector.catalog.Identifier)'
at org.apache.spark.sql.catalyst.analysis.RewriteViewCommands$ResolvedIdent$.unapply(RewriteViewCommands.scala:105)
at org.apache.spark.sql.catalyst.analysis.RewriteViewCommands$$anonfun$apply$1.applyOrElse(RewriteViewCommands.scala:54)
at org.apache.spark.sql.catalyst.analysis.RewriteViewCommands$$anonfun$apply$1.applyOrElse(RewriteViewCommands.scala:50)
at org.apache.spark.sql.catalyst.plans.logical.AnalysisHelper.$anonfun$resolveOperatorsUpWithPruning$3(AnalysisHelper.scala:139)
at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(origin.scala:107)
```

The reason is Spark 4.1 ResolvedIdentifier added a third parameter 'output: Seq[Attribute] = Nil'

Spark 4.0
```
case class ResolvedIdentifier(
catalog: CatalogPlugin,
identifier: Identifier) extends LeafNodeWithoutStats {
override def output: Seq[Attribute] = Nil
}
```

Spark 4.1
```
case class ResolvedIdentifier(
catalog: CatalogPlugin,
identifier: Identifier,
override val output: Seq[Attribute] = Nil) extends LeafNodeWithoutStats

object ResolvedIdentifier {
def unapply(ri: ResolvedIdentifier): Option[(CatalogPlugin, Identifier)] = {
Some((ri.catalog, ri.identifier))
}
}
```

### Willingness to contribute

- [x] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the CREATE OR REPLACE VIEW query against Spark 4.1 and inspect the RewriteViewCommands stack trace. Compare the Spark 4.0 and 4.1 ResolvedIdentifier signatures shown in the issue, then verify that the query succeeds without the NoSuchMethodError after the compatibility fix.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.