apache / apache/hudi

Scheduling compaction discards the extra metadata the plan generator put on the compaction plan

Closed
#19,939 1 comment 0 reactions 0 assignees View on GitHub
area:table-service priority:high
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

### Problem

`BaseHoodieCompactionPlanGenerator` lets a plan generator record its own state in the compaction plan's extra metadata through `getExtraMetadata(...)`. An incremental strategy uses this to remember which partitions it skipped so the next run can pick them up.

Since #18183, `BaseHoodieTableServiceClient.scheduleTableServiceInternal` enriches the caller's extra-metadata option with the Hudi version and engine properties before scheduling, so the option handed to `ScheduleCompactionActionExecutor` is never empty. The executor then applies it with `extraMetadata.ifPresent(plan::setExtraMetadata)`, which replaces the map the generator just produced. The generator's entries are silently dropped on every schedule, and incremental compaction strategies built on this hook regress to full scans without any error.

Verified with a debug probe: the generator computed its metadata correctly and the persisted plan carried only the enrichment keys.

### Expected

The caller-provided entries are merged into the plan's extra metadata rather than overwriting it. Caller entries can win on a key collision; entries only the generator can produce must survive.

### Environment

Hudi master (1.3.0-SNAPSHOT), any engine. Log compaction scheduling shares the code path.

Contributor guide

No contributing guide indexed for this repository

Research direction

Trace scheduling from BaseHoodieTableServiceClient.scheduleTableServiceInternal into ScheduleCompactionActionExecutor, then inspect the metadata produced by BaseHoodieCompactionPlanGenerator. Confirm the scheduling path merges caller-provided entries with generator metadata, with caller values winning collisions, and verify the persisted plan retains generator-only entries.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.