INSERT_OVERWRITE can proceed while clustering is pending on the same file groups
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
**Describe the problem you faced**
When a Hudi table has a pending clustering plan and an `INSERT_OVERWRITE` (or `INSERT_OVERWRITE_TABLE`) operation targets the same partition(s), the operation proceeds and replaces the file groups that clustering was scheduled against. The clustering update strategies (`SparkRejectUpdateStrategy`, `SparkAllowUpdateStrategy`) only inspect explicit *record-level* updates to detect a conflict. `INSERT_OVERWRITE` does not tag records with existing file groups — it declares whole partitions to be replaced wholesale via `getPartitionToReplacedFileIds`. The strategies never see the to-be-replaced groups, so `SparkRejectUpdateStrategy` (the default) does not throw, and the overwrite is admitted. With `hoodie.clustering.rollback.pending.replacecommit=true`, this can also lead to clustering being rolled back repeatedly (pipeline starvation).
**To Reproduce**
1. Configure a table with `hoodie.clustering.updates.strategy=org.apache.hudi.client.clustering.update.strategy.SparkRejectUpdateStrategy` (the default).
2. Ingest some data into partition `p`.
3. Schedule clustering on `p` (do not run it).
4. Issue `INSERT_OVERWRITE` against partition `p`.
5. The overwrite completes; the `Reject` strategy did not detect the conflict.
**Expected behavior**
`SparkRejectUpdateStrategy` should throw `HoodieClusteringUpdateException` because the file groups being replaced overlap with pending clustering. Same expectation for `INSERT_OVERWRITE_TABLE` against any partition that has pending clustering.
**Environment Description**
* Hudi version: master
* Spark version: 3.5
* Storage: any
**Additional context**
`DELETE_PARTITION` already has its own pre-existing check (`DeletePartitionUtils.checkForPendingTableServiceActions`) and is unaffected. PR #18829 addresses this.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with SparkRejectUpdateStrategy and SparkAllowUpdateStrategy, then trace getPartitionToReplacedFileIds and the INSERT_OVERWRITE/INSERT_OVERWRITE_TABLE path. Reproduce with pending clustering on the same partition and verify the reject strategy raises HoodieClusteringUpdateException; compare with the existing DELETE_PARTITION check in DeletePartitionUtils. PR #18829 is noted as addressing this.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 35/100