On failures, rebalancer might leave untracked shards on the workers
Open
enhancement
rebalancer
user_reported
- Dominant language
- C
- Stars
- 12.8k
- Forks
- 794
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 31
Description
We recently had some incidents where there were lots of shards on one of the workers, and none was in the metadata.
Given the structure of the rebalancer, this is possible. The rebalancer does bunch of stuff outside of the distributed transaction and commits (see `SendCommandListToWorkerOutsideTransaction`).
So, if the coordinator itself or one of the last commands that goes to worker fails, we'll have the untracked shards on the workers. These shards could even have the data inside.
Contributor guide
Assessment
This issue has not been assessed yet.