apache / apache/hudi

InsertOverwrite operation on consistent hashing resulting in wrong data

Open
#16,021 0 comments 0 reactions 0 assignees View on GitHub
area:index from-jira priority:high type:bug
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

{code:java}
spark.sql(
s"""insert into $tableName values
|(5, 'a', 35, 1000, '2021-01-05'),
|(1, 'a', 31, 1000, '2021-01-05'),
|(3, 'a', 33, 1000, '2021-01-05'),
|(4, 'b', 16, 1000, '2021-01-05'),
|(2, 'b', 18, 1000, '2021-01-05'),
|(6, 'b', 17, 1000, '2021-01-05'),
|(8, 'a', 21, 1000, '2021-01-05'),
|(9, 'a', 22, 1000, '2021-01-05'),
|(7, 'a', 23, 1000, '2021-01-05')
|""".stripMargin)

// Insert overwrite static partition
spark.sql(
s"""
| insert overwrite table $tableName partition(dt = '2021-01-05')
| select * from (select 13 , 'a2', 12, 1000) limit 10
""".stripMargin)

spark.sql(
s"""
| insert into $tableName values
| (5, 'a3', 35, 1000, '2021-01-05'),
| (3, 'a3', 33, 1000, '2021-01-05')
""".stripMargin)
{code}
After running the above case, we expect the result of the snapshot would be (13, "a3", 12.0, 1000, "2021-01-05"), (5, "a3", 35, 1000, "2021-01-05"), (3, "a3", 33, 1000, "2021-01-05"). 

But the actual result is (13,a2,12.0,1000,2021-01-05).

The root cause is that after running insert overwrite into a consistent bucket index, the  file groups in consistent_hashing_metadata does not match file groups on storage any more.

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-6364
- Type: Bug

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the SQL sequence in the issue, including the static-partition INSERT OVERWRITE and subsequent INSERT INTO. Compare consistent_hashing_metadata with the file groups on storage after the overwrite. Done means the snapshot contains the expected rows, including the later a3 records.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering, databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.