citusdata / citusdata/citus

Repairing shards for Replication Factor > 1 is partially broken on hardware failures

Open
#4,351 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
C
Stars
12.8k
Forks
794
Avg merge
2d 14h
Merged PRs (30d)
31

Description

Scenario:
- User has a distributed table with replication factor = 2.
- There are 2 workers
- One of the workers experience a hardware failure
- All the modification queries are multi-shard queries, meaning that they never mark placements as INVALID, but rather the query fails rather than marking it as invalid.
```SQL
update test set c = 14;
ERROR: terminating connection due to administrator command
CONTEXT: while executing command on localhost:9700
Time: 8.328 ms
```
- The user replaces the hardware.
- The user does `master_disable_node` on the removed node
- The user setups postgres, installs the citus extension on the new hardware
- As none of the placements are marked with SHARD_STATE_INVALID, the user cannot use `master_copy_shard_placement ()` to repair the shard placements on the new hardware.

As a workaround, the user needs to manually mark the placements as INVALID `update pg_dist_shard_placement SET SHARDSTATE=3 WHERE shardid IN (shards in the failed node) and nodeport = failed_port;`

If user does the above, the repair functionality can be used now.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.