apache / apache/hudi

INSERT_OVERWRITE can proceed while clustering is pending on the same file groups

Open
#18,830 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

**Describe the problem you faced**

When a Hudi table has a pending clustering plan and an `INSERT_OVERWRITE` (or `INSERT_OVERWRITE_TABLE`) operation targets the same partition(s), the operation proceeds and replaces the file groups that clustering was scheduled against. The clustering update strategies (`SparkRejectUpdateStrategy`, `SparkAllowUpdateStrategy`) only inspect explicit *record-level* updates to detect a conflict. `INSERT_OVERWRITE` does not tag records with existing file groups — it declares whole partitions to be replaced wholesale via `getPartitionToReplacedFileIds`. The strategies never see the to-be-replaced groups, so `SparkRejectUpdateStrategy` (the default) does not throw, and the overwrite is admitted. With `hoodie.clustering.rollback.pending.replacecommit=true`, this can also lead to clustering being rolled back repeatedly (pipeline starvation).

**To Reproduce**

1. Configure a table with `hoodie.clustering.updates.strategy=org.apache.hudi.client.clustering.update.strategy.SparkRejectUpdateStrategy` (the default).
2. Ingest some data into partition `p`.
3. Schedule clustering on `p` (do not run it).
4. Issue `INSERT_OVERWRITE` against partition `p`.
5. The overwrite completes; the `Reject` strategy did not detect the conflict.

**Expected behavior**

`SparkRejectUpdateStrategy` should throw `HoodieClusteringUpdateException` because the file groups being replaced overlap with pending clustering. Same expectation for `INSERT_OVERWRITE_TABLE` against any partition that has pending clustering.

**Environment Description**

* Hudi version: master
* Spark version: 3.5
* Storage: any

**Additional context**

`DELETE_PARTITION` already has its own pre-existing check (`DeletePartitionUtils.checkForPendingTableServiceActions`) and is unaffected. PR #18829 addresses this.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with SparkRejectUpdateStrategy and SparkAllowUpdateStrategy, then trace getPartitionToReplacedFileIds and the INSERT_OVERWRITE/INSERT_OVERWRITE_TABLE path. Reproduce with pending clustering on the same partition and verify the reject strategy raises HoodieClusteringUpdateException; compare with the existing DELETE_PARTITION check in DeletePartitionUtils. PR #18829 is noted as addressing this.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.