Spark: CREATE VIEW ... WITH SCHEMA EVOLUTION fails for VALUES queries during SELECT
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 132
Description
### Apache Iceberg version
None
### Query engine
Spark
### Please describe the bug 🐞
A view created using WITH SCHEMA EVOLUTION over a VALUES query is created successfully, but fails during query execution with an INCOMPATIBLE_VIEW_SCHEMA_CHANGE exception.
This scenario worked correctly in Spark 3.5 and issue exists in Spark 4.0.
Reproducer
CREATE VIEW q7(c1, c2)
WITH SCHEMA EVOLUTION
AS
VALUES (1, 2);
SELECT * FROM q7;
Actual Result
The view is created successfully, but the SELECT fails with:
`org.apache.spark.sql.AnalysisException: [INCOMPATIBLE_VIEW_SCHEMA_CHANGE] The SQL query of view spark_catalog.q7 has an incompatible schema change and column c1 cannot be resolved. Expected 1 column named c1 but got []`
Expected Result
Since the view definition is based on a constant VALUES clause whose schema cannot evolve, the query should return:
+---+---+
| c1| c2|
+---+---+
| 1| 2|
+---+---+
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
Contributor guide
Research direction
Start by running the supplied CREATE VIEW and SELECT reproducer on Spark 4.0, then compare the result with Spark 3.5. Trace the Spark SQL handling of schema evolution for a VALUES-based view and add coverage for this query; done means SELECT * FROM q7 returns c1=1 and c2=2 without INCOMPATIBLE_VIEW_SCHEMA_CHANGE.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- spark, sql
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100