Repairing shards for Replication Factor > 1 is partially broken on hardware failures
- Dominant language
- C
- Stars
- 12.8k
- Forks
- 794
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 31
Description
Scenario:
- User has a distributed table with replication factor = 2.
- There are 2 workers
- One of the workers experience a hardware failure
- All the modification queries are multi-shard queries, meaning that they never mark placements as INVALID, but rather the query fails rather than marking it as invalid.
```SQL
update test set c = 14;
ERROR: terminating connection due to administrator command
CONTEXT: while executing command on localhost:9700
Time: 8.328 ms
```
- The user replaces the hardware.
- The user does `master_disable_node` on the removed node
- The user setups postgres, installs the citus extension on the new hardware
- As none of the placements are marked with SHARD_STATE_INVALID, the user cannot use `master_copy_shard_placement ()` to repair the shard placements on the new hardware.
As a workaround, the user needs to manually mark the placements as INVALID `update pg_dist_shard_placement SET SHARDSTATE=3 WHERE shardid IN (shards in the failed node) and nodeport = failed_port;`
If user does the above, the repair functionality can be used now.
Contributor guide
Assessment
This issue has not been assessed yet.