[BUG] delete_partition followed by re-insert silently loses data under Non-Blocking Concurrency Control
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
### Bug Description
**What happened:**
Under Non-Blocking Concurrency Control (NBCC), performing a `delete_partition` on a partition and then re-inserting rows into that same partition results in **silent data loss** — the re-inserted rows are written to storage but become invisible to all readers, with no error raised.
This is the same underlying invariant violation as [#19791](https://github.com/apache/hudi/pull/19791) (`insert overwrite` under NBCC), but reached through a different operation pair. #19791 deliberately scoped its fix to `insert overwrite` only; this issue tracks the broader root cause.
**What you expected:**
When a bucket's fixed file group is retired by a replacecommit, subsequent NBCC writes to that bucket should land on a **fresh** file id (e.g. bump the trailing `-0` generation in `newBucketFileIdForNBCC` / the bucket-id-getter) so the new file group id no longer matches the retired one. This fixes both the `insert overwrite` and `delete_partition`+re-insert cases at the source. Needs careful thought on compaction, log-file grouping, and reader-side bucket resolution.
**Steps to reproduce:**
1. Create a MOR + simple-bucket table with `hoodie.write.concurrency.mode = NON_BLOCKING_CONCURRENCY_CONTROL`.
2. Insert some rows into partition `p`.
3. Run `delete_partition` on `p`. This produces a **replacecommit** that records bucket `p`'s fixed file id (`…-0`) as replaced.
4. Re-insert rows into partition `p`. NBCC routes them to the **same** fixed file id.
5. Query the table.
**Expected:** the rows from step 4 are visible.
**Actual:** the rows from step 4 are silently invisible — the file system view still treats that file group id as replaced by the step-3 replacecommit, so it filters the freshly written slice out. No error is raised.
### Environment
**Hudi version:**
**Query engine:** (Spark/Flink/Trino etc)
**Relevant configs:**
### Logs and Stack Trace
_No response_
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.