apache / apache/hudi

[BUG] the deleted data reappeared after clustering on the table

Open
#7,839 9 comments 0 reactions 1 assignee Claimed by @nsivabalan View on GitHub
area:table-service engine:spark issue:data-consistency priority:medium
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

**Environment Description**

* Hudi version :
0.12.2
* Spark version :
3.2.2
* Hadoop version :
2.7.3
* Storage :
hdfs

**Describe the problem you faced**

I have a hudi table and I deleted some records, then I clustered it, finally I found that the deleted data reappeared when I check the result.

**To Reproduce**

1. I have a hudi table called cluster_test and delete some records
```
deldelete from cluster_test where id in (2,8,11);
```
the result after delete is :
企业微信截图_e767a9a0-741c-4d83-b25b-bd1c747bf68a

2. then I submit a cluster job
```
spark-submit --class org.apache.hudi.utilities.HoodieClusteringJob hudi-utilities-bundle_2.12-0.12.2.jar \
--props file:///Users/qishuiqing/develop/hudi/clusteringjob.properties \
--mode scheduleAndExecute --base-path 'hdfs://localhost:9000/user/hive/warehouse/hudi.db/cluster_test' \
--table-name cluster_test --parallelism 4 \
--spark-memory 4g
```
the result after cluster is :
企业微信截图_f9e34400-113c-43e9-9e26-1d3b095b7752

3. table struct
```
col_name data_type comment
_hoodie_commit_time string
_hoodie_commit_seqno string
_hoodie_record_key string
_hoodie_partition_path string
_hoodie_file_name string
id int
name string
ts bigint

# Detailed Table Information
Database hudi
Table cluster_test
Created By Spark 3.2.2
Type EXTERNAL
Provider hudi
Table Properties [preCombineField=ts, primaryKey=id, type=mor]
Statistics 2173911 bytes
Location hdfs://localhost:9000/user/hive/warehouse/hudi.db/cluster_test
Serde Library org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe
InputFormat org.apache.hudi.hadoop.realtime.HoodieParquetRealtimeInputFormat
OutputFormat org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat
```

**conclusion**
this is a sericous bug needed to be fixed

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.