citusdata / citusdata/citus

On failures, rebalancer might leave untracked shards on the workers

Open
#5,329 1 comment 1 reaction 0 assignees View on GitHub
enhancement rebalancer user_reported
Dominant language
C
Stars
12.8k
Forks
794
Avg merge
2d 14h
Merged PRs (30d)
31

Description

We recently had some incidents where there were lots of shards on one of the workers, and none was in the metadata.

Given the structure of the rebalancer, this is possible. The rebalancer does bunch of stuff outside of the distributed transaction and commits (see `SendCommandListToWorkerOutsideTransaction`).

So, if the coordinator itself or one of the last commands that goes to worker fails, we'll have the untracked shards on the workers. These shards could even have the data inside.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.