apache / apache/pinot

Rebalance with Downtime Can Cause Data Loss

Open
#10,788 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
2d 55m
Merged PRs (30d)
182

Description

Taking an example of a realtime table, if the segment download url link for some segments are missing, issuing a Table Rebalance with Downtime can cause data loss with very high likelihood (even with peer download enabled).

I think it is better to check if all segments have the download url link first before starting the Rebalance, and throw an error and skip Rebalance in that case.

We could also improve the Rebalance algorithm to be more aware of which segments are safe for concurrent move operations (i.e. segments with valid deepstore links), and which are not, and perform a rebalance accordingly.

In any state, I don't think the system should be allowed to cause a data loss.

Contributor guide

Open the contributing guide

Research direction

The issue does not name files or tests. Start by reproducing the realtime-table scenario with missing segment download URLs and Table Rebalance with Downtime; done means the rebalance is rejected before starting or safely handles eligible segments without data loss.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.