Remove config of enabling lead controller resource
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 195
Description
Lead controller resource is design to logically separate the per-table functionality in pinot cluster.
As lead controller resource feature has been implemented and running for a while, we'd like to enable this feature to keep the behavior consistent in all Pinot clusters.
Before enabling the lead controller resource, there is only 1 pinot controller (which is also the helix lead controller) that does all the pinot controller periodic tasks and realtime segment completions for all the tables. The lead helix/pinot controller can be found at the `CONTROLLER/LEADER` ZNode.
After enabling the lead controller resource, helix controllers and pinot controllers don't have to run on the same hardware. Znode `CONTROLLER/LEADER` only denotes Helix lead controller. All the controller periodic tasks and realtime segment completions are distributed to all the available pinot controllers. The lead Pinot controller for each of the tables can be found by the following APIs:
1. Get the leaders for all the tables in the cluster
GET /leader/tables
2. Given a table name, return the partition id and lead controller instance id
GET /leader/tables/{tableName}
Offline segment upload won't bind to any of the pinot controllers; all the available pinot controllers accepts segment upload since it's neither controller periodic tasks nor realtime segment completions.
Details can be found in doc: https://cwiki.apache.org/confluence/display/PINOT/Controller+Separation+between+Helix+and+Pinot
There will be two PRs for this issue:
The first PR will enable lead controller in all Pinot clusters. New onboard clusters or existing running clusters that start to run with this code will have lead controller resource enabled.
The second PR will clean up the code that handles the old protocol in the repo, which will be merged in Apache Pinot Release 0.8.0 or higher. The cleanup should be done in the following classes:
- Controller: LeadControllerManager.java
- Server: ControllerLeaderLocator.java
Contributor guide
Research direction
Start with Controller/LeadControllerManager.java and Server/ControllerLeaderLocator.java, then review the merged PR #6725 and the linked controller-separation design document. Identify the old protocol handling that remains after lead controller resource is enabled, and consider the cleanup complete when both classes no longer handle that legacy protocol.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- distributed-systems
- Issue type
- Refactor
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100