Add inorder commit for multi compaction plan
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
When we use the multi plan compaction,we may need the plan to commit in order,so I try to figure that by create a set to store the instant that are not ready,and I transform the compactionPlans(List of Pair) to the sinkFunction.When a commit is ready,I remove the instant from the noReadyCache,and I will check that if it the earliest compaction need to commit, if that commit the compaction instant and check if the next one is ready and commit.if it's not the earliest one, just remove its instant from the noReadyCache.
[https://github.com/apache/hudi/pull/6512]
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-4725
- Type: Improvement
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing the multi-plan compaction flow and the compactionPlans list into the sinkFunction described in the issue. Verify how ready compactions are currently committed; done means commits follow compaction-instant order, including subsequent ready instants.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, stream-processing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100