citusdata / citusdata/citus

Running rebalance_table_shards concurrently with DDL commands causes deadlocks

Open
#1,210 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
C
Stars
12.8k
Forks
794
Avg merge
2d 14h
Merged PRs (30d)
31

Description

(I realize this isn't a complete description and might be difficult to replicate. I've left the cluster running and I'm happy to add logs or give access to anyone who wants to investigate)

`orders` is a table which uses streaming replication, and there are two nodes which have been added with `start_metadata_sync_to_node`.

````
ec2-user=# SELECT rebalance_table_shards('orders', 0);
NOTICE: Moving shard 102851 from 10.192.0.165:5432 to 10.192.0.158:5432 ...
WARNING: deadlock detected
DETAIL: Process 16988 waits for AccessShareLock on relation 24600 of database 16384; blocked by process 16942.
Process 16942 waits for ShareLock on advisory lock [16384,0,102851,4]; blocked by process 16988.
HINT: See server log for query details.
NOTICE: Moving shard 102853 from 10.192.0.165:5432 to 10.192.0.158:5432 ...
NOTICE: Moving shard 102855 from 10.192.0.165:5432 to 10.192.0.158:5432 ...
WARNING: deadlock detected
DETAIL: Process 16991 waits for AccessShareLock on relation 24600 of database 16384; blocked by process 16942.
Process 16942 waits for ShareLock on advisory lock [16384,0,102855,4]; blocked by process 16991.
HINT: See server log for query details.
rebalance_table_shards
------------------------

(1 row)
```

In another session on the master I run:

```
ec2-user=# ALTER TABLE orders ADD COLUMN test_col int;
NOTICE: using one-phase commit for distributed DDL commands
HINT: You can enable two-phase commit for extra safety with: SET citus.multi_shard_commit_protocol TO '2pc'
ALTER TABLE
ec2-user=# ALTER TABLE orders DROP COLUMN test_col;
ALTER TABLE
```

Those complete and the metadata successfully syncs, but they cause the deadlocks found above.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.