apache / apache/hudi

CompactionAdminClient.unscheduleCompactionFileId drops every other operation in the same partition

Open Beginner friendly
#19,881 1 comment 0 reactions 0 assignees View on GitHub
type:bug
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 4h
Merged PRs (30d)
112

Description

### Bug Description

**What happened:**
`CompactionAdminClient.unscheduleCompactionFileId` removes more operations from the compaction plan than the one file group it was asked to unschedule. The filter at master 6ac27904, `CompactionAdminClient.java:149-150`, is

```java
plan.getOperations().stream().filter(op ->
(!op.getFileId().equals(fgId.getFileId())) && (!op.getPartitionPath().equals(fgId.getPartitionPath())))
```

which keeps only the operations that differ in both file id and partition path. Every other pending operation in the same partition is dropped from the rewritten plan, and its file groups are silently never compacted by that plan. The intended condition is `!(sameFileId && samePartition)`.

Reachable from the `compaction unscheduleFileId` command in hudi-cli (through `SparkMain.doCompactUnscheduleFile`) and from any direct caller of `CompactionAdminClient`.

**What you expected:**
Only the operation for the given file group is removed; the other operations in the same partition stay in the plan.

**Steps to reproduce:**
1. MOR table with a pending compaction plan holding two operations in the same partition (file groups A and B).
2. `unscheduleCompactionFileId(fgId(A), skipValidation=true, dryRun=false)`.
3. Reload the plan: B is gone as well.

Found while writing `TestCompactionCommand.testSparkMainCompactUnscheduleFile` in #19877; that test only asserts the target file group is gone so it passes with and without the bug. A regression test belongs in `TestCompactionAdminClient` next to the fix.

Related, same class: `unscheduleCompactionPlan` ignores its `skipValidation` and `parallelism` arguments and `repairCompaction` ignores `dryRun` and discards its validation result, so `compaction repair` cannot repair anything. Worth deciding whether those arguments should be removed or the log-file rename logic restored.

### Environment

**Hudi version:** master (6ac27904), also present in 1.0.x and 0.15.x
**Query engine:** hudi-cli / any CompactionAdminClient caller
**Relevant configs:** none

### Logs and Stack Trace

No exception; the rewritten `.compaction.requested` plan is missing operations.

### Related

- #15573: the same `compaction repair` / `validate` / `unschedule` commands fail on S3 because the `.ser` output path is resolved against the wrong file system. Different defect, same code path; a fix touching these commands could address both.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in CompactionAdminClient.java:149-150, reached from SparkMain.doCompactUnscheduleFile, and inspect the existing unschedule logic. Add a regression case to TestCompactionAdminClient using two operations in one partition, then run that test. Done means unscheduling file group A removes A while retaining file group B in the rewritten plan.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
88/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.