apache / apache/hudi

[SUPPORT] Does spark.sql("MERGE INTO") supports schema evolution write option

Open
#8,502 3 comments 0 reactions 0 assignees View on GitHub
area:sql priority:high type:feature
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

**Describe the problem you faced**
I have created a table with 2 columns namely `userId` and `updatedAt`. Now I'm passing new column `nested` in the `merge into` command but gotten an exception.

```java
spark.sql("" +
"MERGE INTO target USING source ON target.userId = source.userId " +
"WHEN MATCHED THEN UPDATE SET target.nested = struct(source.colA), target.updatedAt = source.updatedAt " +
"WHEN NOT MATCHED THEN INSERT (userId, nested, updatedAt) " +
"VALUES (source.userId, struct(source.colA), source.updatedAt)" +
"")
```
Error
```
Cannot resolve 'target.nested`
```

**To Reproduce**

Steps to reproduce the behavior:

1. Merge into command with new column specified.
2. Try setting `.config("hoodie.schema.on.read.enable", "true")` doesn't help.

**Expected behavior**
The schema should evolve and detect that this is a new column.

**Environment Description**

* Hudi version : 0.12.2

* Spark version : 3.3.1

* Hive version : -

* Hadoop version : -

* Storage (HDFS/S3/GCS..) : -

* Running on Docker? (yes/no) : -

**Additional context**

Add any other context about the problem here.

**Stacktrace**

```Add the stacktrace of the error.```

Contributor guide

No contributing guide indexed for this repository

Research direction

No source file or test is named. Start with the MERGE INTO reproduction on Hudi 0.12.2 and Spark 3.3.1, then check whether hoodie.schema.on.read.enable is relevant to schema evolution during MERGE. Done means establishing a tested resolution for adding nested without the unresolved target.nested error.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark, sql
Domain
data-engineering, databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.