[SUPPORT] - Records deleted with via "hard delete" appear after next commit
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
**_Tips before filing an issue_**
- Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)?
- Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.
- If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly.
**Describe the problem you faced**
We have a COW table running in Hudi 0.14.0 (before it was running on Hudi 0.11.1) some of the table properties are the following:
- Table updated using Glue 4.0
- Update method: USERT
- Index: BLOOM INDEX
- Partition Keys: year, month, day
- Maximum files before archiving: 15
- Minimum files after archive: 11
- Commits before clean: 10
We noticed some duplicates for a given partition, for instance, 2025-01-23. This should not be happening since we are performing an Upsert operation, however we are yet to find a root cause for this unexpected behaviour.
I have identified the duplicated records and followed the steps described [here](https://hudi.apache.org/docs/next/writing_data#deletes) to perform **hard deletes**.
Even though the records are deleted at first, immediately after a new commit is done to the table, they reappear, even when the data upserted to the table doesn't contain the previously deleted ones.
**To Reproduce**
Steps to reproduce the behavior:
1. Have a table which is updated via UPSERT, with BLOOM_INDEX. In our particular scenario, duplicates were introduced within the same partition (unknown root cause).
2. Drop some records from the table
3. Make a regular commit.
4. Dropped Records should reappear (at least in our experience)
**Expected behavior**
Deleted records should not reappear after a new update to the table is made.
Is there something that I might be missing?
How can I ensure that the deleted records do not reappear?
**Environment Description**
* Hudi version : 0.14.0
* Glue Version: 4.0
* Spark version : 3.3
* Hive version : The version in Glue 4.0
* Hadoop version : The version in Glue 4.0
* Storage (HDFS/S3/GCS..) : S3
* Running on Docker? (yes/no) : No
**Additional context**
Add any other context about the problem here.
**Stacktrace**
```Add the stacktrace of the error.```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the behavior with Hudi 0.14.0 on Glue 4.0 and Spark 3.3 using a COW table, UPSERT, a Bloom index, hard deletes, and a subsequent commit. Done means identifying why deleted records reappear and confirming that they remain absent after the next update; the issue provides no named source files or tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, java, spark
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100