apache / apache/hudi

[BUG] delete_partition followed by re-insert silently loses data under Non-Blocking Concurrency Control

Open
#19,793 1 comment 0 reactions 1 assignee Claimed by @zhang-arvin View on GitHub
type:bug
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

### Bug Description

**What happened:**

Under Non-Blocking Concurrency Control (NBCC), performing a `delete_partition` on a partition and then re-inserting rows into that same partition results in **silent data loss** — the re-inserted rows are written to storage but become invisible to all readers, with no error raised.

This is the same underlying invariant violation as [#19791](https://github.com/apache/hudi/pull/19791) (`insert overwrite` under NBCC), but reached through a different operation pair. #19791 deliberately scoped its fix to `insert overwrite` only; this issue tracks the broader root cause.

**What you expected:**

When a bucket's fixed file group is retired by a replacecommit, subsequent NBCC writes to that bucket should land on a **fresh** file id (e.g. bump the trailing `-0` generation in `newBucketFileIdForNBCC` / the bucket-id-getter) so the new file group id no longer matches the retired one. This fixes both the `insert overwrite` and `delete_partition`+re-insert cases at the source. Needs careful thought on compaction, log-file grouping, and reader-side bucket resolution.

**Steps to reproduce:**
1. Create a MOR + simple-bucket table with `hoodie.write.concurrency.mode = NON_BLOCKING_CONCURRENCY_CONTROL`.
2. Insert some rows into partition `p`.
3. Run `delete_partition` on `p`. This produces a **replacecommit** that records bucket `p`'s fixed file id (`…-0`) as replaced.
4. Re-insert rows into partition `p`. NBCC routes them to the **same** fixed file id.
5. Query the table.

**Expected:** the rows from step 4 are visible.
**Actual:** the rows from step 4 are silently invisible — the file system view still treats that file group id as replaced by the step-3 replacecommit, so it filters the freshly written slice out. No error is raised.

### Environment

**Hudi version:**
**Query engine:** (Spark/Flink/Trino etc)
**Relevant configs:**

### Logs and Stack Trace

_No response_

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.