[SUPPORT] Does spark.sql("MERGE INTO") supports schema evolution write option
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
**Describe the problem you faced**
I have created a table with 2 columns namely `userId` and `updatedAt`. Now I'm passing new column `nested` in the `merge into` command but gotten an exception.
```java
spark.sql("" +
"MERGE INTO target USING source ON target.userId = source.userId " +
"WHEN MATCHED THEN UPDATE SET target.nested = struct(source.colA), target.updatedAt = source.updatedAt " +
"WHEN NOT MATCHED THEN INSERT (userId, nested, updatedAt) " +
"VALUES (source.userId, struct(source.colA), source.updatedAt)" +
"")
```
Error
```
Cannot resolve 'target.nested`
```
**To Reproduce**
Steps to reproduce the behavior:
1. Merge into command with new column specified.
2. Try setting `.config("hoodie.schema.on.read.enable", "true")` doesn't help.
**Expected behavior**
The schema should evolve and detect that this is a new column.
**Environment Description**
* Hudi version : 0.12.2
* Spark version : 3.3.1
* Hive version : -
* Hadoop version : -
* Storage (HDFS/S3/GCS..) : -
* Running on Docker? (yes/no) : -
**Additional context**
Add any other context about the problem here.
**Stacktrace**
```Add the stacktrace of the error.```
Contributor guide
No contributing guide indexed for this repository
Research direction
No source file or test is named. Start with the MERGE INTO reproduction on Hudi 0.12.2 and Spark 3.3.1, then check whether hoodie.schema.on.read.enable is relevant to schema evolution during MERGE. Done means establishing a tested resolution for adding nested without the unresolved target.nested error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark, sql
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100