Citus 12.1 stopping rebalance_table_shards(drain_only := true) causes duplicate shards and needs manual intervention
- Dominant language
- C
- Stars
- 12.8k
- Forks
- 794
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 31
Description
Context: removing nodes from cluster. by first draining them. nodes are marked as shouldhaveshards=false, i am logged in on one of the coordinators.
After starting the drain i wanted to update citus.max_background_task_executors_per_node so i canceled the drain.
on restart is got the following error:
```
ERROR: shard move failed as the orphaned shard public._120737 leftover from the previous move could not be cleaned up
```
When checking the shards i see:
```
table_name | shardid | shard_name | citus_table_type | colocation_id | nodename | nodeport | shard_size
----------------------+---------+-----------------------------+------------------+---------------+--------------+----------+------------
| 120737 | _120737 | distributed | 2 | 10.1.1.33 | 5432 | 735838208
| 120737 | _120737 | distributed | 2 | 10.1.1.75 | 5432 | 735838208
```
33 and 75 are primary and replica for a worker node.
```
citus=# select * from pg_dist_cleanup;
record_id | operation_id | object_type | object_name | node_group_id | policy_type
-----------+--------------+-------------+---------------------------------------+---------------+-------------
782755 | 8714 | 1 | public._127393 | 2 | 1
782756 | 8714 | 1 | public._127649 | 2 | 1
782764 | 8714 | 1 | public._129697 | 2 | 1
(3 rows)
citus=# SELECT * FROM rebalance_table_shards(drain_only := true);
NOTICE: Moving shard 120737 from 10.1.1.33:5432 to 10.1.1.87:5432 ...
ERROR: shard move failed as the orphaned shard public.activities_127393 leftover from the previous move could not be cleaned up
CONTEXT: while executing command on localhost:5432
citus=# select * from pg_dist_cleanup;
record_id | operation_id | object_type | object_name | node_group_id | policy_type
-----------+--------------+-------------+---------------------------------------+---------------+-------------
782755 | 8714 | 1 | public._127393 | 2 | 1
782756 | 8714 | 1 | public._127649 | 2 | 1
(2 rows)
citus=# delete from pg_dist_cleanup;
DELETE 2
citus=# SELECT * FROM rebalance_table_shards(drain_only := true);
NOTICE: Moving shard 120737 from 10.1.1.33:5432 to 10.1.1.87:5432 ...
```
87 is the primary of a not disabled worker node.
I restarted the rebalance by cleaning up the pg_dist_cleanup as referenced in issue/#7333
-----
Questions:
Is this expected behavior? Should the cleanup make it possible to restart the rebalance if i gave it more time? or is intervention required.
Contributor guide
Assessment
This issue has not been assessed yet.