element-hq / element-hq/synapse

Synapse causes deadlock when racing server key lookup

Open
#19,401 0 comments 0 reactions 0 assignees View on GitHub
A-Database A-Validation O-Occasional S-Major T-Defect
Dominant language
Python
Stars
4.6k
Forks
600
Avg merge
5d 22h
Merged PRs (30d)
51

Description

### Description

When synapse is trying to look up signing keys and has multiple notaries configured, it queries them all at the same time. If at least two of them also respond at exactly the same time synapse causes a deadlock in postgres while trying to insert the key

### Steps to reproduce

- configure more than one trusted key server
- do stuff that makes synapse lookup keys from notaries
- wait

### Homeserver

codestorm.net

### Synapse Version

1.145

### Installation Method

Debian packages from packages.matrix.org

### Database

PostgreSQL

### Workers

Multiple workers

### Platform

Synapse in a Debian 12 LXC on proxmox VE 9

### Configuration

_No response_

### Relevant log output

```shell
2026-01-21 23:58:29.000 Matrix-Codestorm
Matrix-Codestorm postgres[834]: [186-2] 2026-01-21 23:58:29.563 CET [834] synapse_user@synapse STATEMENT: INSERT INTO server_signature_keys (server_name, key_id, from_server, ts_added_ms, ts_valid_until_ms, verify_key) VALUES ('que.trolling.win','ed25519:b4YBNxcB','t2bot.io',1769036309557,1769510486670,'\x5b6f79da20df7f60e1eec905a9df150f07fe37932e1690ec6c128b6720b282f5'::bytea),('que.trolling.win','ed25519:SqIxvAyv','t2bot.io',1769036309557,1768905686670,'\xe9f7274c28733244ae2acf5de1a8767b0c8c787b595900fa9330f487216d25d8'::bytea),('que.trolling.win','ed25519:Zwarh37L','t2bot.io',1769036309557,1768905686670,'\x83081b496bfc570849a7ab41b9cfe9d856864130a9e3eeb0ae5a8ff5db3df539'::bytea),('que.trolling.win','ed25519:vLle2JLh','t2bot.io',1769036309557,1768905686670,'\x2c5d7816baff13336768e1f38c877f481c5173f108043182a3a7b3fd3df5d9c8'::bytea) ON CONFLICT (server_name, key_id) DO UPDATE SET from_server=EXCLUDED.from_server,

2026-01-21 23:58:29.000 Matrix-Codestorm
Matrix-Codestorm postgres[846]: [194-2] 2026-01-21 23:58:29.563 CET [846] synapse_user@synapse STATEMENT: INSERT INTO server_signature_keys (server_name, key_id, from_server, ts_added_ms, ts_valid_until_ms, verify_key) VALUES ('que.trolling.win','ed25519:Zwarh37L','aguiarvieira.pt',1769036309562,1768609351319,'\x83081b496bfc570849a7ab41b9cfe9d856864130a9e3eeb0ae5a8ff5db3df539'::bytea),('que.trolling.win','ed25519:SqIxvAyv','aguiarvieira.pt',1769036309562,1768004551319,'\xe9f7274c28733244ae2acf5de1a8767b0c8c787b595900fa9330f487216d25d8'::bytea),('que.trolling.win','ed25519:vLle2JLh','aguiarvieira.pt',1769036309562,1768004551319,'\x2c5d7816baff13336768e1f38c877f481c5173f108043182a3a7b3fd3df5d9c8'::bytea) ON CONFLICT (server_name, key_id) DO UPDATE SET from_server=EXCLUDED.from_server, ts_added_ms=EXCLUDED.ts_added_ms, ts_valid_until_ms=EXCLUDED.ts_valid_until_ms, verify_key=EXCLUDED.verify_key

2026-01-21 23:58:30.000 Matrix-Codestorm
Matrix-Codestorm postgres[846]: [195-1] 2026-01-21 23:58:30.564 CET [846] synapse_user@synapse ERROR: deadlock detected

2026-01-21 23:58:30.000 Matrix-Codestorm
Matrix-Codestorm postgres[846]: [195-2] 2026-01-21 23:58:30.564 CET [846] synapse_user@synapse DETAIL: Process 846 waits for ShareLock on transaction 83682579; blocked by process 834.

2026-01-21 23:58:30.000 Matrix-Codestorm
Matrix-Codestorm matrix-synapse-client_worker1[601]: 2026-01-21 23:58:30,564 - synapse.storage.txn - 827 - WARNING - PerspectivesKeyFetcher-327 - [TXN OPERROR] {store_server_keys_response-1617c2} deadlock detected

2026-01-21 23:58:30.000 Matrix-Codestorm
Matrix-Codestorm matrix-synapse-client_worker1[601]: DETAIL: Process 846 waits for ShareLock on transaction 83682579; blocked by process 834.

2026-01-21 23:58:30.000 Matrix-Codestorm
Matrix-Codestorm matrix-synapse-client_worker1[601]: Process 834 waits for ShareLock on transaction 83682580; blocked by process 846.

2026-01-21 23:58:30.000 Matrix-Codestorm
Matrix-Codestorm matrix-synapse-client_worker1[601]: HINT: See server log for query details.

2026-01-21 23:58:30.000 Matrix-Codestorm
Matrix-Codestorm matrix-synapse-client_worker1[601]: CONTEXT: while inserting index tuple (722,28) in relation "server_signature_keys"

2026-01-21 23:58:30.000 Matrix-Codestorm
Matrix-Codestorm postgres[846]: [195-3] #011Process 834 waits for ShareLock on transaction 83682580; blocked by process 846.

2026-01-21 23:58:30.000 Matrix-Codestorm
Matrix-Codestorm postgres[846]: [195-7] 2026-01-21 23:58:30.564 CET [846] synapse_user@synapse HINT: See server log for query details.

2026-01-21 23:58:30.000 Matrix-Codestorm
Matrix-Codestorm postgres[846]: [195-8] 2026-01-21 23:58:30.564 CET [846] synapse_user@synapse CONTEXT: while inserting index tuple (722,28) in relation "server_signature_keys"
```

### Anything else that would be useful to know?

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.