Merge into not get the right value in update action
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
{{val incDF = Seq(
HudiDataWithData(1, "lb", 8, "shu", 1646643412l),
HudiDataWithData(1, "lb", 2, "shu", 1646643412l),
HudiDataWithData(1, "lb", 7, "shu", 1646643412l),
HudiDataWithData(2, "gy", 12, "shu", 1646643193l),
HudiDataWithData(1, "cc", 22, "wei", 1646643193l),
HudiDataWithData(2, "xy", 23, "wei", 1646643193l)
).toDF
incDF.createOrReplaceTempView("inc_table")
spark.sql(
s"""
|create table hudi_cow_pt_tbl (
| id int,
| name string,
| data int,
| country string,
| ts bigint
|) using hudi
|tblproperties (
| type = 'cow',
| primaryKey = 'id',
| preCombineField = 'ts'
| )
|partitioned by (country)
|location 'D:/tmp/hudi_data/hudi_merge_test01'
|""".stripMargin)
spark.sql(
s"""
|merge into hudi_cow_pt_tbl as target
|using (
| select id, name, data, country, ts from inc_table
|) source
|on source.id = target.id
|when matched and source.data > target.data then
|update set target.data = source.data, target.ts = source.ts
|when not matched then
|insert *
|""".stripMargin)}}
h2. A clear and concise description of the problem.
when we query, record with 'lb' is 7 not 8,so it is a bug?
h2. [20230308151658945,20230308151658945_0_0,1,country=wei,c0d7ce20-c40f-4064-8e9c-22d4dd2b1e2a-0_0-14-48_20230309094305111.parquet,1,cc,44,1646643193,wei]
h2. [20230308151658945,20230308151658945_0_1,2,country=wei,c0d7ce20-c40f-4064-8e9c-22d4dd2b1e2a-0_0-14-48_20230309094305111.parquet,2,xy,46,1646643193,wei]
h2. [20230308214428068,20230308214428068_1_0,2,country=shu,28617435-0f34-4c3f-a9e1-859611b68094-0_1-14-49_20230309094305111.parquet,2,gy,12,1646643193,shu]
h2. [20230309094305111,20230309094305111_1_1,1,country=shu,28617435-0f34-4c3f-a9e1-859611b68094-0_1-14-49_20230309094305111.parquet,1,lb,7,1646643412,shu]
{{}}
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-5908
- Type: Bug
- Epic: https://issues.apache.org/jira/browse/HUDI-3431
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by running the supplied Spark SQL MERGE reproduction against a Hudi COW table and inspect the update-action path for the duplicate id=1 records in partition country=shu. Done means the matched record for name 'lb' has data 8 rather than 7, with the relevant regression coverage added around this reproduction.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala, spark, sql
- Domain
- data, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100