apache / apache/hudi

Incremental clean planning should ignore partitions from instants with only new file groups

Open
#18,014 0 comments 0 reactions 0 assignees View on GitHub
type:devtask
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

### Task Description

**What needs to be done:**
Currently when incremental clean planner scans all instants since latest earliest commit to retain (ECTR), `org.apache.hudi.table.action.clean.CleanPlanner#getPartitionsForInstants` will add all partitions across all instants' `partitionToWriteStats`. We should optimize this flow such that when processing `commit` instants we only add a partition if the `commit` metadata has any entry in `partitionToWriteStats` where a file group was updated. If any instant updated/replaced a file group, we should still add it to list of partitions to scan (so that we don't "miss" any files to clean).

**Why this task is needed:*
For insert-only workloads (where small file handling is disabled), if there are thousands of partitions touched by `commit`s since the latest ECTR, then clean planner will have to unnecessarily scan all of these partitions (even though there is nothing to clean in these partitions).

### Task Type

Performance optimization

### Related Issues

**Parent feature issue:** (if applicable )
**Related issues:**
NOTE: Use `Relationships` button to add parent/blocking issues after issue is created.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in org.apache.hudi.table.action.clean.CleanPlanner#getPartitionsForInstants and trace how commit metadata populates partitionToWriteStats. Verify the planner distinguishes commits with only new file groups from instants that updated or replaced file groups, while still scanning every partition that may contain files to clean.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Refactor
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.