apache / apache/hudi

Merge into not get the right value in update action

Open
#15,837 0 comments 0 reactions 0 assignees View on GitHub
area:sql from-jira priority:blocker type:bug
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

{{val incDF = Seq(
HudiDataWithData(1, "lb", 8, "shu", 1646643412l),
HudiDataWithData(1, "lb", 2, "shu", 1646643412l),
HudiDataWithData(1, "lb", 7, "shu", 1646643412l),
HudiDataWithData(2, "gy", 12, "shu", 1646643193l),
HudiDataWithData(1, "cc", 22, "wei", 1646643193l),
HudiDataWithData(2, "xy", 23, "wei", 1646643193l)
).toDF
incDF.createOrReplaceTempView("inc_table")

spark.sql(
s"""
|create table hudi_cow_pt_tbl (
| id int,
| name string,
| data int,
| country string,
| ts bigint
|) using hudi
|tblproperties (
| type = 'cow',
| primaryKey = 'id',
| preCombineField = 'ts'
| )
|partitioned by (country)
|location 'D:/tmp/hudi_data/hudi_merge_test01'
|""".stripMargin)

spark.sql(
s"""
|merge into hudi_cow_pt_tbl as target
|using (
| select id, name, data, country, ts from inc_table
|) source
|on source.id = target.id
|when matched and source.data > target.data then
|update set target.data = source.data, target.ts = source.ts
|when not matched then
|insert *
|""".stripMargin)}}
h2. A clear and concise description of the problem.
when we query, record with 'lb' is 7 not 8,so it is a bug?
h2. [20230308151658945,20230308151658945_0_0,1,country=wei,c0d7ce20-c40f-4064-8e9c-22d4dd2b1e2a-0_0-14-48_20230309094305111.parquet,1,cc,44,1646643193,wei]
h2. [20230308151658945,20230308151658945_0_1,2,country=wei,c0d7ce20-c40f-4064-8e9c-22d4dd2b1e2a-0_0-14-48_20230309094305111.parquet,2,xy,46,1646643193,wei]
h2. [20230308214428068,20230308214428068_1_0,2,country=shu,28617435-0f34-4c3f-a9e1-859611b68094-0_1-14-49_20230309094305111.parquet,2,gy,12,1646643193,shu]
h2. [20230309094305111,20230309094305111_1_1,1,country=shu,28617435-0f34-4c3f-a9e1-859611b68094-0_1-14-49_20230309094305111.parquet,1,lb,7,1646643412,shu]

{{}}

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-5908
- Type: Bug
- Epic: https://issues.apache.org/jira/browse/HUDI-3431

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by running the supplied Spark SQL MERGE reproduction against a Hudi COW table and inspect the update-action path for the duplicate id=1 records in partition country=shu. Done means the matched record for name 'lb' has data 8 rather than 7, with the relevant regression coverage added around this reproduction.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, spark, sql
Domain
data, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.