apache / apache/iceberg

Spark: CREATE VIEW ... WITH SCHEMA EVOLUTION fails for VALUES queries during SELECT

Open
#17,288 3 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 11h
Merged PRs (30d)
132

Description

### Apache Iceberg version

None

### Query engine

Spark

### Please describe the bug 🐞

A view created using WITH SCHEMA EVOLUTION over a VALUES query is created successfully, but fails during query execution with an INCOMPATIBLE_VIEW_SCHEMA_CHANGE exception.
This scenario worked correctly in Spark 3.5 and issue exists in Spark 4.0.
Reproducer
CREATE VIEW q7(c1, c2)
WITH SCHEMA EVOLUTION
AS
VALUES (1, 2);

SELECT * FROM q7;
Actual Result

The view is created successfully, but the SELECT fails with:
`org.apache.spark.sql.AnalysisException: [INCOMPATIBLE_VIEW_SCHEMA_CHANGE] The SQL query of view spark_catalog.q7 has an incompatible schema change and column c1 cannot be resolved. Expected 1 column named c1 but got []`

Expected Result

Since the view definition is based on a constant VALUES clause whose schema cannot evolve, the query should return:

+---+---+
| c1| c2|
+---+---+
| 1| 2|
+---+---+

### Willingness to contribute

- [ ] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time

Contributor guide

Open the contributing guide

Research direction

Start by running the supplied CREATE VIEW and SELECT reproducer on Spark 4.0, then compare the result with Spark 3.5. Trace the Spark SQL handling of schema evolution for a VALUES-based view and add coverage for this query; done means SELECT * FROM q7 returns c1=1 and c2=2 without INCOMPATIBLE_VIEW_SCHEMA_CHANGE.

Written by the indexing model from the issue text.

Assessment

Tech stack
spark, sql
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.