apache / apache/pinot

Minion Realtime To Offline Segment Task runs slow as there is no parallelism per task

Open
#10,854 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
189

Description

1 minion host running 1 real-time to offline task only allows 1 subtask to run at a time. Since there are no parallelism, minion cannot scale if the realtime table becomes too huge.

Note: Mapper phase is usually the bottleneck as observed from my logs.

Contributor guide

Open the contributing guide

Research direction

Start by tracing the Minion Realtime To Offline Segment Task and its Mapper phase, using the reported one-subtask-at-a-time behavior as the entry point. Determine where task parallelism is constrained and define done as allowing useful parallel subtasks so a large realtime table can scale better.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data, distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.