apache / apache/pinot

Remove config of enabling lead controller resource

Open
#6,730 0 comments 0 reactions 0 assignees View on GitHub
release-notes
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 3h
Merged PRs (30d)
195

Description

Lead controller resource is design to logically separate the per-table functionality in pinot cluster.

As lead controller resource feature has been implemented and running for a while, we'd like to enable this feature to keep the behavior consistent in all Pinot clusters.

Before enabling the lead controller resource, there is only 1 pinot controller (which is also the helix lead controller) that does all the pinot controller periodic tasks and realtime segment completions for all the tables. The lead helix/pinot controller can be found at the `CONTROLLER/LEADER` ZNode.

After enabling the lead controller resource, helix controllers and pinot controllers don't have to run on the same hardware. Znode `CONTROLLER/LEADER` only denotes Helix lead controller. All the controller periodic tasks and realtime segment completions are distributed to all the available pinot controllers. The lead Pinot controller for each of the tables can be found by the following APIs:
1. Get the leaders for all the tables in the cluster
GET /leader/tables

2. Given a table name, return the partition id and lead controller instance id
GET /leader/tables/{tableName}

Offline segment upload won't bind to any of the pinot controllers; all the available pinot controllers accepts segment upload since it's neither controller periodic tasks nor realtime segment completions.

Details can be found in doc: https://cwiki.apache.org/confluence/display/PINOT/Controller+Separation+between+Helix+and+Pinot

There will be two PRs for this issue:
The first PR will enable lead controller in all Pinot clusters. New onboard clusters or existing running clusters that start to run with this code will have lead controller resource enabled.
The second PR will clean up the code that handles the old protocol in the repo, which will be merged in Apache Pinot Release 0.8.0 or higher. The cleanup should be done in the following classes:

- Controller: LeadControllerManager.java
- Server: ControllerLeaderLocator.java

Contributor guide

Open the contributing guide

Research direction

Start with Controller/LeadControllerManager.java and Server/ControllerLeaderLocator.java, then review the merged PR #6725 and the linked controller-separation design document. Identify the old protocol handling that remains after lead controller resource is enabled, and consider the cleanup complete when both classes no longer handle that legacy protocol.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems
Issue type
Refactor
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.