[SUPPORT] After upgrading hudi 0.14.1, use Spark SQL merge into to update the matched_action, the case of the column name and the expression name does not match, resulting in an exception.
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
****_Tips before filing an issue_**
- Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)?
- Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.
- If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly.
**Describe the problem you faced**
Hello team, after **upgrading **from** hudi0.13.1** to **hudi0.14.1**, I found that a large number of merge into tasks failed. After a simple test, it is concluded that the case of column name and expression name is inconsistent in the `merge into ... WHEN MATCHED THEN UPDATE SET ` operation.
[Not sure if it has anything to do with this new feature](https://github.com/apache/hudi/pull/9083/)
**To Reproduce**
Steps to reproduce the behavior:
1. Create test tables and data
```sql
create table merge_source (
id int, name string, price double
) using hudi
tblproperties
(primaryKey = 'id');
insert into merge_source values (1, "old_a1", 22.22), (2, "new_a2", 33.33), (3, "new_a3", 44.44);
create table hudi_cow_tbl_v14 (
id INT,
name STRING,
price DOUBLE
) USING hudi
tblproperties
(primaryKey = 'id');
insert into hudi_cow_tbl_v14 values (1, "oldid1", 100.00), (2, "oldid2", 200.00);
```
2. Execute a merge statement
```sql
merge into hudi_cow_tbl_v14 as target
using merge_source as source
on target.id = source.id
when matched then update set ID=source.ID, name=source.name
```

3. Results of using arthars debug watch methods
```properties
method=org.apache.spark.sql.hudi.analysis.HoodieSpark32PlusResolveReferences.resolveMergeExprOrFail location=AtExceptionExit
ts=2024-01-24 15:06:18; [cost=1.207048ms] result=@ArrayList[
@Object[][
@UnresolvedAttribute['ID],
@Project[Project
+- SubqueryAlias target
+- SubqueryAlias spark_catalog.test.hudi_cow_tbl_v14
+- Relation test.hudi_cow_tbl_v14[
_hoodie_commit_time#608,_hoodie_commit_seqno#609,
_hoodie_record_key#610,_hoodie_partition_path#611,
_hoodie_file_name#612,id#613,name#614,price#615]
BaseFileOnlyRelation(org.apache.spark.sql.SQLContext@22decfa2,HoodieTableMetaClient
{basePath='hdfs://xx/user/hive/warehouse/test.db/hudi_cow_tbl_v14',
metaPath='hdfs://xx/user/hive/warehouse/test.db/hudi_cow_tbl_v14/.hoodie',
tableType=COPY_ON_WRITE},
Map(hoodie.embed.timeline.server -> true,
hoodie.datasource.hive_sync.mode -> hms,
hoodie.insert.shuffle.parallelism -> 20,
hoodie.datasource.hive_sync.support_timestamp -> true,
hoodie.schema.on.read.enable -> true,
path -> hdfs://xxx/user/hive/warehouse/test.db/hudi_cow_tbl_v14,
hoodie.filesystem.view.type -> EMBEDDED_KV_STORE,
hoodie.datasource.hive_sync.metastore.uris -> thrift://xxx,
hoodie.delete.shuffle.parallelism -> 20,
hoodie.datasource.query.type -> snapshot,
hoodie.datasource.write.schema.allow.auto.evolution.column.drop -> true,
hoodie.compact.inline -> false,
hoodie.datasource.write.reconcile.schema -> true,
hoodie.upsert.shuffle.parallelism -> 20,
hoodie.query.as.ro.table -> false,
hoodie.bulkinsert.shuffle.parallelism -> 20)
,None,List(),None)
],
],
@HoodieSpark32PlusResolveReferences[
MatchMergeIntoTable$module=@MatchMergeIntoTable$[org.apache.spark.sql.hudi.analysis.HoodieSpark32PlusResolveReferences$MatchMergeIntoTable$@41623694],
spark=@SparkSession[org.apache.spark.sql.SparkSession@b1e63b5],
sparkAdapter=@Spark3_3Adapter[org.apache.spark.sql.adapter.Spark3_3Adapter@46d83ec4],
bitmap$0=@Boolean[true],
ruleId=null,
ruleName=@String[org.apache.spark.sql.hudi.analysis.HoodieSpark32PlusResolveReferences],
org$apache$spark$internal$Logging$$log_=null,
bitmap$0=@Boolean[false],
],
null,
]
```
**Expected behavior**
A clear and concise description of what you expected to happen.
**Environment Description**
- Hudi version : 0.14.1
- Spark version : 3.3.2
- Hive version : 3.1.2
- Hadoop version : 3.3.3
- Storage (HDFS/S3/GCS..) : HDFS
- Running on Docker? (yes/no) : no
**Additional context**
Add any other context about the problem here.
**Stacktrace**
```Add the stacktrace of the error.```
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.