apache / apache/druid

BroadcastDistributionRules to load segments on whole tiers rather than making server-by-server decisions

Open
#7,382 4 comments 0 reactions 0 assignees View on GitHub
Area - Segment Balancing/Coordination Contributions Welcome Design Review
Dominant language
Java
Stars
14.1k
Forks
3.8k
Avg merge
2d 58m
Merged PRs (30d)
233

Description

Broadcast rules which load segments on servers with colocated data sources (server-by-server decision) are very hard or impossible to implement correctly. I think the problems of non-atomic moves during balancing are underestimated.

When rules are loaded from the metadata store, Coordinator can figure out load rules for data sources and maps them to "colocated data sources", obtaining a list of tiers on which broadcasted segments should be effectively loaded. Then it loads broadcasted segments on all servers in the needed tiers, as described here: https://github.com/apache/incubator-druid/pull/4077#issuecomment-293732924

FYI @jihoonson @gianm

Contributor guide

Open the contributing guide

Research direction

Start with the linked pull request discussion and the Coordinator path that reads load rules from the metadata store. Trace how colocated data sources are mapped to tiers and define how whole-tier loading should behave during balancing; done means broadcasted segments load on every server in each required tier without server-by-server decisions.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.