BroadcastDistributionRules to load segments on whole tiers rather than making server-by-server decisions
- Dominant language
- Java
- Stars
- 14.1k
- Forks
- 3.8k
- Avg merge
- 2d 58m
- Merged PRs (30d)
- 233
Description
Broadcast rules which load segments on servers with colocated data sources (server-by-server decision) are very hard or impossible to implement correctly. I think the problems of non-atomic moves during balancing are underestimated.
When rules are loaded from the metadata store, Coordinator can figure out load rules for data sources and maps them to "colocated data sources", obtaining a list of tiers on which broadcasted segments should be effectively loaded. Then it loads broadcasted segments on all servers in the needed tiers, as described here: https://github.com/apache/incubator-druid/pull/4077#issuecomment-293732924
FYI @jihoonson @gianm
Contributor guide
Research direction
Start with the linked pull request discussion and the Coordinator path that reads load rules from the metadata store. Trace how colocated data sources are mapped to tiers and define how whole-tier loading should behave during balancing; done means broadcasted segments load on every server in each required tier without server-by-server decisions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100