[SUPPORT]0.12.3 upgrade to 0.14.0 data duplication
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
**Describe the problem you faced**
For a Hudi (0.12.3) table with existing data, upgrade to 0.14.0. After the upgrade, it is found that the data is duplicated. Check whether the old data file still exists and has not been converted to the new data file. What do I need to do now?

**To Reproduce**
Steps to reproduce the behavior:
1.0.12.3 hudi table
2. upgrade to 0.14.0
3. SELECT * FROM hudi_table_changes('bi_ods_real.online_offline_event_batch_rt', 'latest_state', 'earliest') where id ='6c60e6270272ea8499sygv';
4. data duplication , not be merged
**Expected behavior**
A clear and concise description of what you expected to happen.
**Environment Description**
* Hudi version :0.14.0
* Spark version :3.2.4
* Hive version :3.1.3
* Hadoop version :3.2.2
* Storage (HDFS/S3/GCS..) :s3
* Running on Docker? (yes/no) :no
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the upgrade from Hudi 0.12.3 to 0.14.0 using the existing table on S3 and the reported Spark 3.2.4 environment. Inspect the hudi_table_changes('bi_ods_real.online_offline_event_batch_rt', 'latest_state', 'earliest') query and compare the old and new data files for the reported id. Done means identifying why the records are duplicated and documenting a verified corrective path.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- hadoop, java, spark
- Domain
- data-engineering, databases, stream-processing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100