apache / apache/hudi

[SUPPORT] Duplicate data in base file of MOR table

Open
#10,882 15 comments 0 reactions 0 assignees View on GitHub
issue:data-consistency priority:critical
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

I want to upgrade Hudi from 0.11.1 to 0.13.1, but I encountered the problem of duplicate data. I have never encountered it before with the same configuration.

I have synchronized many tables, and the probability of this happening is very small, which is very strange.

Here is an example of repeated data:
image

hadoop:2.7.7
spark:3.2.0
hudi:0.13.1
index_type: Simple_Bucket

write config:

#Tue Feb 20 15:37:13 CST 2024
hoodie.table.timeline.timezone=LOCAL
hoodie.table.keygenerator.class=org.apache.hudi.keygen.ComplexKeyGenerator
hoodie.table.precombine.field=_binlog_offset
hoodie.table.version=5
hoodie.database.name=
hoodie.datasource.write.hive_style_partitioning=true
hoodie.partition.metafile.use.base.format=false
hoodie.archivelog.folder=archived
hoodie.table.cdc.enabled=false
hoodie.table.name=cffincontractdb_channel_contract_ext_rt
hoodie.compaction.payload.class=org.apache.hudi.common.model.EventTimeAvroPayload
hoodie.populate.meta.fields=true
hoodie.compaction.record.merger.strategy=eeb8d96f-b1e4-49fd-bbf8-28ac514178e5
hoodie.table.type=MERGE_ON_READ
hoodie.datasource.write.partitionpath.urlencode=false
hoodie.table.base.file.format=PARQUET
hoodie.datasource.write.drop.partition.columns=false
hoodie.timeline.layout.version=1
hoodie.table.partition.fields=_db,_tbl,_id_mod
hoodie.table.recordkey.fields=id
hoodie.table.checksum=1185151018

hoodie.datasource.write.operation=UPSERT
hoodie.payload.event.time.field=_binlog_offset

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the duplicate-data case during the 0.11.1-to-0.13.1 upgrade using the supplied Hadoop, Spark, MOR, Simple_Bucket, and write configuration. Compare the affected base files and table timeline across versions; done means identifying a reproducible cause or the configuration and processing step that introduces duplicates.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.