Support automatic schema evolution for SQL MERGE INTO with UPDATE */ INESRT *
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
I tried using MERGE INTO with UPDATE * and INSERT * statement with full schema evolution enabled.
I noticed that during insert new columns from incoming batch (that do not exist in target table yet) are dropped and target schema is applied. No warnings nor failed writes.
Therefore can we as users automatically evolve schema on MERGE INTO operations?
I guess this should only be supported when we use update set * and insert * in merge operation.
*Expected behavior*
When incoming data is missing columns that already declared in target table these should be injected with default/null values.
When incoming data has new columns that are not yet declared in the target table, these should be added to the target table.
Case when incoming data has both missing columns and new columns, missing columns should be injected with null/ default values, new columns should be added to the target table.
New columns should be reflected in metastore table schema.
Should support complex types, and nested schemas.
Currently similar thing is supported for dataframe writes if both schema reconciliation and schema evolution configs are enabled, see HUDI-4276.
From user experience perspective it would be easier if I had _mergeSchema_ (as for parquet spark datasource) config to enable this feature for both spark sql and df write.
Thread from dev mailing list as a reference:
[https://lists.apache.org/thread/kr59hh7yqr2c1y33kzfv3n97h6ydbz9b]
GH issue: [https://github.com/apache/hudi/issues/5899]
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-4872
- Type: Improvement
- Epic: https://issues.apache.org/jira/browse/HUDI-1297
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reviewing HUDI-4276 and the linked mailing-list thread, then trace the MERGE INTO path for UPDATE * and INSERT *. Compare it with the existing dataframe schema reconciliation and schema evolution behavior. Done means missing columns are filled with null or defaults, new and nested columns are added to the target and reflected in the metastore, with coverage for SQL and dataframe writes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark, sql
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100