apache / apache/pinot

Big increase in latency around the time the new server joins the cluster and starts serving queries

Open
#8,247 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 3h
Merged PRs (30d)
195

Description

The following image is from a server getting killed, then restarting and rejoining the cluster a minute later.

The graph shows QPS

![image](https://user-images.githubusercontent.com/89811563/155618197-2bb077cb-a00e-4ba4-b434-ce56d7cb2612.png)

and here's latency:
![image](https://user-images.githubusercontent.com/89811563/155618347-829166b3-13a1-48af-8e3a-cb99e3fbc10c.png)

Latency spikes to timeouts around the time the server dies and then once again once the server starts serving queries.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the reported server kill, restart, and cluster rejoin sequence while comparing the QPS and latency graphs attached to the issue. Done means identifying the cause of the timeout spikes and verifying a fix; the issue names no source file, entry point, or test.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.