Scheduling compaction discards the extra metadata the plan generator put on the compaction plan
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
### Problem
`BaseHoodieCompactionPlanGenerator` lets a plan generator record its own state in the compaction plan's extra metadata through `getExtraMetadata(...)`. An incremental strategy uses this to remember which partitions it skipped so the next run can pick them up.
Since #18183, `BaseHoodieTableServiceClient.scheduleTableServiceInternal` enriches the caller's extra-metadata option with the Hudi version and engine properties before scheduling, so the option handed to `ScheduleCompactionActionExecutor` is never empty. The executor then applies it with `extraMetadata.ifPresent(plan::setExtraMetadata)`, which replaces the map the generator just produced. The generator's entries are silently dropped on every schedule, and incremental compaction strategies built on this hook regress to full scans without any error.
Verified with a debug probe: the generator computed its metadata correctly and the persisted plan carried only the enrichment keys.
### Expected
The caller-provided entries are merged into the plan's extra metadata rather than overwriting it. Caller entries can win on a key collision; entries only the generator can produce must survive.
### Environment
Hudi master (1.3.0-SNAPSHOT), any engine. Log compaction scheduling shares the code path.
Contributor guide
No contributing guide indexed for this repository
Research direction
Trace scheduling from BaseHoodieTableServiceClient.scheduleTableServiceInternal into ScheduleCompactionActionExecutor, then inspect the metadata produced by BaseHoodieCompactionPlanGenerator. Confirm the scheduling path merges caller-provided entries with generator metadata, with caller values winning collisions, and verify the persisted plan retains generator-only entries.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 72/100