apache / apache/pinot

Lack of documentation about cluster networking

Open
#5,859 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
189

Description

While setting up a Pinot cluster in an environment with strict egress/ingress rules it has been a challenge to bump into various cluster interop that wasn't documented.

Based on the docs I recall, the only concise indication of the connectivity is this diagram: https://docs.pinot.apache.org/basics/architecture#core-components

This came with a couple surprises so far:
* Servers connect to the Controller's API for pulling locally stored segments
* Controllers proxy queries from the query console to the Broker's API

There's other areas that I'm lightly aware of but have not yet dealt with, such as batch ingestion jobs connecting to Controllers for notifying of new segments.

Contributor guide

Open the contributing guide

Research direction

Start with the cluster architecture page at docs.pinot.apache.org/basics/architecture#core-components and review the connectivity examples in this issue. Document the relevant cluster network paths, including server-to-Controller segment access, Controller-to-Broker query proxying, and batch-ingestion connections to Controllers.

Written by the indexing model from the issue text.

Assessment

Domain
distributed-systems, documentation, networking
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.