apache / apache/hudi

[SUPPORT]0.12.3 upgrade to 0.14.0 data duplication

Open
#10,407 12 comments 0 reactions 0 assignees View on GitHub
issue:data-consistency priority:critical
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

**Describe the problem you faced**
For a Hudi (0.12.3) table with existing data, upgrade to 0.14.0. After the upgrade, it is found that the data is duplicated. Check whether the old data file still exists and has not been converted to the new data file. What do I need to do now?

![image](https://github.com/apache/hudi/assets/15028279/e49dd934-f2a8-4596-b051-7a3e4e455721)

**To Reproduce**

Steps to reproduce the behavior:

1.0.12.3 hudi table
2. upgrade to 0.14.0
3. SELECT * FROM hudi_table_changes('bi_ods_real.online_offline_event_batch_rt', 'latest_state', 'earliest') where id ='6c60e6270272ea8499sygv';
4. data duplication , not be merged

**Expected behavior**

A clear and concise description of what you expected to happen.

**Environment Description**

* Hudi version :0.14.0

* Spark version :3.2.4

* Hive version :3.1.3

* Hadoop version :3.2.2

* Storage (HDFS/S3/GCS..) :s3

* Running on Docker? (yes/no) :no

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the upgrade from Hudi 0.12.3 to 0.14.0 using the existing table on S3 and the reported Spark 3.2.4 environment. Inspect the hudi_table_changes('bi_ods_real.online_offline_event_batch_rt', 'latest_state', 'earliest') query and compare the old and new data files for the reported id. Done means identifying why the records are duplicated and documenting a verified corrective path.

Written by the indexing model from the issue text.

Assessment

Tech stack
hadoop, java, spark
Domain
data-engineering, databases, stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.