Rebalance with Downtime Can Cause Data Loss
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 2d 55m
- Merged PRs (30d)
- 182
Description
Taking an example of a realtime table, if the segment download url link for some segments are missing, issuing a Table Rebalance with Downtime can cause data loss with very high likelihood (even with peer download enabled).
I think it is better to check if all segments have the download url link first before starting the Rebalance, and throw an error and skip Rebalance in that case.
We could also improve the Rebalance algorithm to be more aware of which segments are safe for concurrent move operations (i.e. segments with valid deepstore links), and which are not, and perform a rebalance accordingly.
In any state, I don't think the system should be allowed to cause a data loss.
Contributor guide
Research direction
The issue does not name files or tests. Start by reproducing the realtime-table scenario with missing segment download URLs and Table Rebalance with Downtime; done means the rebalance is rejected before starting or safely handles eligible segments without data loss.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100