element-hq / element-hq/synapse
Synapse causes deadlock when racing server key lookup
- Dominant language
- Python
- Stars
- 4.6k
- Forks
- 600
- Avg merge
- 5d 22h
- Merged PRs (30d)
- 51
Description
### Description
When synapse is trying to look up signing keys and has multiple notaries configured, it queries them all at the same time. If at least two of them also respond at exactly the same time synapse causes a deadlock in postgres while trying to insert the key
### Steps to reproduce
- configure more than one trusted key server
- do stuff that makes synapse lookup keys from notaries
- wait
### Homeserver
codestorm.net
### Synapse Version
1.145
### Installation Method
Debian packages from packages.matrix.org
### Database
PostgreSQL
### Workers
Multiple workers
### Platform
Synapse in a Debian 12 LXC on proxmox VE 9
### Configuration
_No response_
### Relevant log output
```shell
2026-01-21 23:58:29.000 Matrix-Codestorm
Matrix-Codestorm postgres[834]: [186-2] 2026-01-21 23:58:29.563 CET [834] synapse_user@synapse STATEMENT: INSERT INTO server_signature_keys (server_name, key_id, from_server, ts_added_ms, ts_valid_until_ms, verify_key) VALUES ('que.trolling.win','ed25519:b4YBNxcB','t2bot.io',1769036309557,1769510486670,'\x5b6f79da20df7f60e1eec905a9df150f07fe37932e1690ec6c128b6720b282f5'::bytea),('que.trolling.win','ed25519:SqIxvAyv','t2bot.io',1769036309557,1768905686670,'\xe9f7274c28733244ae2acf5de1a8767b0c8c787b595900fa9330f487216d25d8'::bytea),('que.trolling.win','ed25519:Zwarh37L','t2bot.io',1769036309557,1768905686670,'\x83081b496bfc570849a7ab41b9cfe9d856864130a9e3eeb0ae5a8ff5db3df539'::bytea),('que.trolling.win','ed25519:vLle2JLh','t2bot.io',1769036309557,1768905686670,'\x2c5d7816baff13336768e1f38c877f481c5173f108043182a3a7b3fd3df5d9c8'::bytea) ON CONFLICT (server_name, key_id) DO UPDATE SET from_server=EXCLUDED.from_server,
2026-01-21 23:58:29.000 Matrix-Codestorm
Matrix-Codestorm postgres[846]: [194-2] 2026-01-21 23:58:29.563 CET [846] synapse_user@synapse STATEMENT: INSERT INTO server_signature_keys (server_name, key_id, from_server, ts_added_ms, ts_valid_until_ms, verify_key) VALUES ('que.trolling.win','ed25519:Zwarh37L','aguiarvieira.pt',1769036309562,1768609351319,'\x83081b496bfc570849a7ab41b9cfe9d856864130a9e3eeb0ae5a8ff5db3df539'::bytea),('que.trolling.win','ed25519:SqIxvAyv','aguiarvieira.pt',1769036309562,1768004551319,'\xe9f7274c28733244ae2acf5de1a8767b0c8c787b595900fa9330f487216d25d8'::bytea),('que.trolling.win','ed25519:vLle2JLh','aguiarvieira.pt',1769036309562,1768004551319,'\x2c5d7816baff13336768e1f38c877f481c5173f108043182a3a7b3fd3df5d9c8'::bytea) ON CONFLICT (server_name, key_id) DO UPDATE SET from_server=EXCLUDED.from_server, ts_added_ms=EXCLUDED.ts_added_ms, ts_valid_until_ms=EXCLUDED.ts_valid_until_ms, verify_key=EXCLUDED.verify_key
2026-01-21 23:58:30.000 Matrix-Codestorm
Matrix-Codestorm postgres[846]: [195-1] 2026-01-21 23:58:30.564 CET [846] synapse_user@synapse ERROR: deadlock detected
2026-01-21 23:58:30.000 Matrix-Codestorm
Matrix-Codestorm postgres[846]: [195-2] 2026-01-21 23:58:30.564 CET [846] synapse_user@synapse DETAIL: Process 846 waits for ShareLock on transaction 83682579; blocked by process 834.
2026-01-21 23:58:30.000 Matrix-Codestorm
Matrix-Codestorm matrix-synapse-client_worker1[601]: 2026-01-21 23:58:30,564 - synapse.storage.txn - 827 - WARNING - PerspectivesKeyFetcher-327 - [TXN OPERROR] {store_server_keys_response-1617c2} deadlock detected
2026-01-21 23:58:30.000 Matrix-Codestorm
Matrix-Codestorm matrix-synapse-client_worker1[601]: DETAIL: Process 846 waits for ShareLock on transaction 83682579; blocked by process 834.
2026-01-21 23:58:30.000 Matrix-Codestorm
Matrix-Codestorm matrix-synapse-client_worker1[601]: Process 834 waits for ShareLock on transaction 83682580; blocked by process 846.
2026-01-21 23:58:30.000 Matrix-Codestorm
Matrix-Codestorm matrix-synapse-client_worker1[601]: HINT: See server log for query details.
2026-01-21 23:58:30.000 Matrix-Codestorm
Matrix-Codestorm matrix-synapse-client_worker1[601]: CONTEXT: while inserting index tuple (722,28) in relation "server_signature_keys"
2026-01-21 23:58:30.000 Matrix-Codestorm
Matrix-Codestorm postgres[846]: [195-3] #011Process 834 waits for ShareLock on transaction 83682580; blocked by process 846.
2026-01-21 23:58:30.000 Matrix-Codestorm
Matrix-Codestorm postgres[846]: [195-7] 2026-01-21 23:58:30.564 CET [846] synapse_user@synapse HINT: See server log for query details.
2026-01-21 23:58:30.000 Matrix-Codestorm
Matrix-Codestorm postgres[846]: [195-8] 2026-01-21 23:58:30.564 CET [846] synapse_user@synapse CONTEXT: while inserting index tuple (722,28) in relation "server_signature_keys"
```
### Anything else that would be useful to know?
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.