apache / apache/hudi

Add inorder commit for multi compaction plan

Open
#15,392 0 comments 0 reactions 0 assignees View on GitHub
area:table-service component:compaction from-jira priority:high type:improvement
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

When we use the multi plan compaction,we may need the plan to commit in order,so I try to figure that by create a set to store the instant that are not ready,and I transform the compactionPlans(List of Pair) to the sinkFunction.When a commit is ready,I remove the instant from the noReadyCache,and I will check that if it the earliest compaction need to commit, if that commit the compaction instant and check if the next one is ready and commit.if it's not the earliest one, just remove its instant from the noReadyCache.

 

[https://github.com/apache/hudi/pull/6512]

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-4725
- Type: Improvement

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing the multi-plan compaction flow and the compactionPlans list into the sinkFunction described in the issue. Verify how ready compactions are currently committed; done means commits follow compaction-instant order, including subsequent ready instants.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, stream-processing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.