Running rebalance_table_shards concurrently with DDL commands causes deadlocks
- Dominant language
- C
- Stars
- 12.8k
- Forks
- 794
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 31
Description
(I realize this isn't a complete description and might be difficult to replicate. I've left the cluster running and I'm happy to add logs or give access to anyone who wants to investigate)
`orders` is a table which uses streaming replication, and there are two nodes which have been added with `start_metadata_sync_to_node`.
````
ec2-user=# SELECT rebalance_table_shards('orders', 0);
NOTICE: Moving shard 102851 from 10.192.0.165:5432 to 10.192.0.158:5432 ...
WARNING: deadlock detected
DETAIL: Process 16988 waits for AccessShareLock on relation 24600 of database 16384; blocked by process 16942.
Process 16942 waits for ShareLock on advisory lock [16384,0,102851,4]; blocked by process 16988.
HINT: See server log for query details.
NOTICE: Moving shard 102853 from 10.192.0.165:5432 to 10.192.0.158:5432 ...
NOTICE: Moving shard 102855 from 10.192.0.165:5432 to 10.192.0.158:5432 ...
WARNING: deadlock detected
DETAIL: Process 16991 waits for AccessShareLock on relation 24600 of database 16384; blocked by process 16942.
Process 16942 waits for ShareLock on advisory lock [16384,0,102855,4]; blocked by process 16991.
HINT: See server log for query details.
rebalance_table_shards
------------------------
(1 row)
```
In another session on the master I run:
```
ec2-user=# ALTER TABLE orders ADD COLUMN test_col int;
NOTICE: using one-phase commit for distributed DDL commands
HINT: You can enable two-phase commit for extra safety with: SET citus.multi_shard_commit_protocol TO '2pc'
ALTER TABLE
ec2-user=# ALTER TABLE orders DROP COLUMN test_col;
ALTER TABLE
```
Those complete and the metadata successfully syncs, but they cause the deadlocks found above.
Contributor guide
Assessment
This issue has not been assessed yet.