Allow more than 1 compaction to be run concurrently in deltastreamer after MOR Incremental read is fully supported
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
Only 1 compaction is run by deltastreamer. Once incremental MOR is supported, we can allow concurrent compaction
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-157
- Type: Task
---
## Comments
22/Jan/21 14:27;shivnarayan;[~vbalaji]: is this something user-reported or more of an enhancement. also, do we need to tag this with "user-support-issues" label? ;;;
Contributor guide
No contributing guide indexed for this repository
Research direction
No file, test, or entry point is named. Start by tracing how DeltaStreamer currently schedules compaction and review the MOR incremental-read support referenced in the issue; the work is done when multiple compactions can run concurrently after incremental MOR reads are fully supported, with appropriate coverage for that behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, stream-processing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100