apache / apache/hudi

Hudi MIT doesn't support the target table reference in extra MATCHED condition

Open
#17,303 1 comment 0 reactions 1 assignee Claimed by @yihua View on GitHub
area:sql from-jira priority:blocker type:devtask
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

Below MIT query fails with Hudi - 

```{{{{}}{}}}

{{MERGE INTO hudi_table_mor_2_partition_columns t
USING comprehensive_merge_source s
ON t.id = s.id
WHEN MATCHED
AND s.operation = 'UPDATE_DEPT_MATCH'
AND t.department = s.department
THEN
UPDATE SET *
WHEN MATCHED
AND s.operation = 'UPDATE_SALARY' THEN
UPDATE SET * ;}}

{{```}}

{{{}if we just remove `{}}}{{{}t.department = s.department{}}}{{{}` from above query it works. It also works if we just remove reference for target table from the same. {}}}{{{}`{}}}{{{}t.department = 'A'{}}}{{{}` doesn't work but {}}}{{{}`s{}}}{{{}.department = 'A'{}}}{{{}` works well.{}}}

 

GIthub issue have reproducible code - [https://github.com/apache/hudi/issues/12180]

Confirmed that DELTA supports these kind of MIT queries but Hudi fails with the Exception.  Also checked with master code and issue still exists.

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-8457
- Type: Sub-task
- Parent: https://issues.apache.org/jira/browse/HUDI-9109
- Fix version(s):
- 1.1.0

---

## Comments

05/Jan/25 23:40;daviszhang;[https://github.com/apache/hudi/issues/12180] the issue has some initial triage;;;

---

06/Jan/25 19:08;yihua;We should mention the constraints of MATCHED conditions as a limitation of MERGE INTO in Hudi Spark SQL, for now.  We should triage how much work it takes to actually support such matching condition.;;;

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.