citusdata / citusdata/citus

Citus doesn't delete shard clean-up records for the node when removing it

Open
#7,333 2 comments 3 reactions 0 assignees View on GitHub
bug
Dominant language
C
Stars
12.8k
Forks
794
Avg merge
2d 14h
Merged PRs (30d)
31

Description

Hi!

Some time ago we lost one of the worker nodes due to hardware failure, we managed to remove the node and rebalance the cluster, however, we are now experiencing degraded cluster performance and share access logs/exclusive access logs piling up. We managed to isolate that ongoing issue to one specific shard and tried copying it over to other worker node. That has failed with the following error message.
```
ERROR: shard move failed as the orphaned shard public.match_potentialmatch_102008 leftover from the previous move could not be cleaned up
```
It still contains ~15GB of data
```
db=# SELECT shardid, table_name, shard_size
FROM citus_shards where shardid = 102008;
shardid | table_name | shard_size
---------+----------------------+-------------
102008 | match_potentialmatch | 14195769344
(1 row)
```
We can also see this shard is scheduled for cleanup. It seems to still be associated with the `node_group` that was lost due to hardware failture and no longer exist in the `pg_dist_node` table.
```
db=# select * from pg_dist_cleanup limit 100;
record_id | operation_id | object_type | object_name | node_group_id | policy_type
-----------+--------------+-------------+------------------------------------+---------------+-------------
315 | 53 | 1 | public.match_potentialmatch_102008 | 4 | 2
(1 row)
```
Screenshot 2023-11-09 at 11 20 24 AM

We couldn't find an answer in the documentation what to do is such situation. Any assistance would be appreciated. Thanks!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.