[SUPPORT] Duplicate data in base file of MOR table
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
I want to upgrade Hudi from 0.11.1 to 0.13.1, but I encountered the problem of duplicate data. I have never encountered it before with the same configuration.
I have synchronized many tables, and the probability of this happening is very small, which is very strange.
Here is an example of repeated data:
hadoop:2.7.7
spark:3.2.0
hudi:0.13.1
index_type: Simple_Bucket
write config:
#Tue Feb 20 15:37:13 CST 2024
hoodie.table.timeline.timezone=LOCAL
hoodie.table.keygenerator.class=org.apache.hudi.keygen.ComplexKeyGenerator
hoodie.table.precombine.field=_binlog_offset
hoodie.table.version=5
hoodie.database.name=
hoodie.datasource.write.hive_style_partitioning=true
hoodie.partition.metafile.use.base.format=false
hoodie.archivelog.folder=archived
hoodie.table.cdc.enabled=false
hoodie.table.name=cffincontractdb_channel_contract_ext_rt
hoodie.compaction.payload.class=org.apache.hudi.common.model.EventTimeAvroPayload
hoodie.populate.meta.fields=true
hoodie.compaction.record.merger.strategy=eeb8d96f-b1e4-49fd-bbf8-28ac514178e5
hoodie.table.type=MERGE_ON_READ
hoodie.datasource.write.partitionpath.urlencode=false
hoodie.table.base.file.format=PARQUET
hoodie.datasource.write.drop.partition.columns=false
hoodie.timeline.layout.version=1
hoodie.table.partition.fields=_db,_tbl,_id_mod
hoodie.table.recordkey.fields=id
hoodie.table.checksum=1185151018
hoodie.datasource.write.operation=UPSERT
hoodie.payload.event.time.field=_binlog_offset
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the duplicate-data case during the 0.11.1-to-0.13.1 upgrade using the supplied Hadoop, Spark, MOR, Simple_Bucket, and write configuration. Compare the affected base files and table timeline across versions; done means identifying a reproducible cause or the configuration and processing step that introduces duplicates.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100