CompactionAdminClient.unscheduleCompactionFileId drops every other operation in the same partition
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 112
Description
### Bug Description
**What happened:**
`CompactionAdminClient.unscheduleCompactionFileId` removes more operations from the compaction plan than the one file group it was asked to unschedule. The filter at master 6ac27904, `CompactionAdminClient.java:149-150`, is
```java
plan.getOperations().stream().filter(op ->
(!op.getFileId().equals(fgId.getFileId())) && (!op.getPartitionPath().equals(fgId.getPartitionPath())))
```
which keeps only the operations that differ in both file id and partition path. Every other pending operation in the same partition is dropped from the rewritten plan, and its file groups are silently never compacted by that plan. The intended condition is `!(sameFileId && samePartition)`.
Reachable from the `compaction unscheduleFileId` command in hudi-cli (through `SparkMain.doCompactUnscheduleFile`) and from any direct caller of `CompactionAdminClient`.
**What you expected:**
Only the operation for the given file group is removed; the other operations in the same partition stay in the plan.
**Steps to reproduce:**
1. MOR table with a pending compaction plan holding two operations in the same partition (file groups A and B).
2. `unscheduleCompactionFileId(fgId(A), skipValidation=true, dryRun=false)`.
3. Reload the plan: B is gone as well.
Found while writing `TestCompactionCommand.testSparkMainCompactUnscheduleFile` in #19877; that test only asserts the target file group is gone so it passes with and without the bug. A regression test belongs in `TestCompactionAdminClient` next to the fix.
Related, same class: `unscheduleCompactionPlan` ignores its `skipValidation` and `parallelism` arguments and `repairCompaction` ignores `dryRun` and discards its validation result, so `compaction repair` cannot repair anything. Worth deciding whether those arguments should be removed or the log-file rename logic restored.
### Environment
**Hudi version:** master (6ac27904), also present in 1.0.x and 0.15.x
**Query engine:** hudi-cli / any CompactionAdminClient caller
**Relevant configs:** none
### Logs and Stack Trace
No exception; the rewritten `.compaction.requested` plan is missing operations.
### Related
- #15573: the same `compaction repair` / `validate` / `unschedule` commands fail on S3 because the `.ser` output path is resolved against the wrong file system. Different defect, same code path; a fix touching these commands could address both.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in CompactionAdminClient.java:149-150, reached from SparkMain.doCompactUnscheduleFile, and inspect the existing unschedule logic. Add a regression case to TestCompactionAdminClient using two operations in one partition, then run that test. Done means unscheduling file group A removes A while retaining file group B in the rewritten plan.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 88/100