Citus doesn't delete shard clean-up records for the node when removing it
- Dominant language
- C
- Stars
- 12.8k
- Forks
- 794
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 31
Description
Hi!
Some time ago we lost one of the worker nodes due to hardware failure, we managed to remove the node and rebalance the cluster, however, we are now experiencing degraded cluster performance and share access logs/exclusive access logs piling up. We managed to isolate that ongoing issue to one specific shard and tried copying it over to other worker node. That has failed with the following error message.
```
ERROR: shard move failed as the orphaned shard public.match_potentialmatch_102008 leftover from the previous move could not be cleaned up
```
It still contains ~15GB of data
```
db=# SELECT shardid, table_name, shard_size
FROM citus_shards where shardid = 102008;
shardid | table_name | shard_size
---------+----------------------+-------------
102008 | match_potentialmatch | 14195769344
(1 row)
```
We can also see this shard is scheduled for cleanup. It seems to still be associated with the `node_group` that was lost due to hardware failture and no longer exist in the `pg_dist_node` table.
```
db=# select * from pg_dist_cleanup limit 100;
record_id | operation_id | object_type | object_name | node_group_id | policy_type
-----------+--------------+-------------+------------------------------------+---------------+-------------
315 | 53 | 1 | public.match_potentialmatch_102008 | 4 | 2
(1 row)
```
We couldn't find an answer in the documentation what to do is such situation. Any assistance would be appreciated. Thanks!
Contributor guide
Assessment
This issue has not been assessed yet.