apache / apache/hudi

[SUPPORT] - Records deleted with via "hard delete" appear after next commit

Open
#12,833 4 comments 0 reactions 0 assignees View on GitHub
issue:data-consistency priority:critical
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

**_Tips before filing an issue_**

- Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)?

- Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.

- If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly.

**Describe the problem you faced**

We have a COW table running in Hudi 0.14.0 (before it was running on Hudi 0.11.1) some of the table properties are the following:
- Table updated using Glue 4.0
- Update method: USERT
- Index: BLOOM INDEX
- Partition Keys: year, month, day
- Maximum files before archiving: 15
- Minimum files after archive: 11
- Commits before clean: 10

We noticed some duplicates for a given partition, for instance, 2025-01-23. This should not be happening since we are performing an Upsert operation, however we are yet to find a root cause for this unexpected behaviour.

I have identified the duplicated records and followed the steps described [here](https://hudi.apache.org/docs/next/writing_data#deletes) to perform **hard deletes**.

Even though the records are deleted at first, immediately after a new commit is done to the table, they reappear, even when the data upserted to the table doesn't contain the previously deleted ones.

**To Reproduce**

Steps to reproduce the behavior:

1. Have a table which is updated via UPSERT, with BLOOM_INDEX. In our particular scenario, duplicates were introduced within the same partition (unknown root cause).
2. Drop some records from the table
3. Make a regular commit.
4. Dropped Records should reappear (at least in our experience)

**Expected behavior**

Deleted records should not reappear after a new update to the table is made.

Is there something that I might be missing?
How can I ensure that the deleted records do not reappear?

**Environment Description**

* Hudi version : 0.14.0

* Glue Version: 4.0

* Spark version : 3.3

* Hive version : The version in Glue 4.0

* Hadoop version : The version in Glue 4.0

* Storage (HDFS/S3/GCS..) : S3

* Running on Docker? (yes/no) : No

**Additional context**

Add any other context about the problem here.

**Stacktrace**

```Add the stacktrace of the error.```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the behavior with Hudi 0.14.0 on Glue 4.0 and Spark 3.3 using a COW table, UPSERT, a Bloom index, hard deletes, and a subsequent commit. Done means identifying why deleted records reappear and confirming that they remain absent after the next update; the issue provides no named source files or tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, java, spark
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.