Minion Realtime To Offline Segment Task runs slow as there is no parallelism per task
Open
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 189
Description
1 minion host running 1 real-time to offline task only allows 1 subtask to run at a time. Since there are no parallelism, minion cannot scale if the realtime table becomes too huge.
Note: Mapper phase is usually the bottleneck as observed from my logs.
Contributor guide
Research direction
Start by tracing the Minion Realtime To Offline Segment Task and its Mapper phase, using the reported one-subtask-at-a-time behavior as the entry point. Determine where task parallelism is constrained and define done as allowing useful parallel subtasks so a large realtime table can scale better.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data, distributed-systems
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100