element-hq / element-hq/synapse
Synapse 1.157 regressed to completely block DB i/o with ReplicationFederationSendEduRestServlet operations
- Dominant language
- Python
- Stars
- 4.6k
- Forks
- 600
- Avg merge
- 5d 22h
- Merged PRs (30d)
- 51
Description
### Description
Upon opening my laptop, Synapse `ReplicationFederationSendEduRestServlet` on my `stream_writer` worker (via [MDAD](https://github.com/spantaleev/matrix-docker-ansible-deploy)) goes crazy, hogs 1500%+ "DB usage" while still accumulating requests.
somehow the federation reader worker also seems involved.
### Steps to reproduce
- open nheko on laptop
- close laptop
- next morning, open laptop
after killing nheko, it takes my server ~4h to recover on its own, as depicted in the graph above. the number of requests continues to rise.
workaround is to restart synapse.
### Homeserver
selfhosted
### Synapse Version
1.157.2
### Installation Method
Other (please mention below)
### Database
single psql (Postgres). ported from original SQLite 5 or so years ago.
### Workers
Multiple workers
### Platform
Official Docker image in docker on Debian on VPS
### Configuration
tried disabling presence, but no luck. using one presence worker via [MDAD](https://github.com/spantaleev/matrix-docker-ansible-deploy).
using `synapse_http_antispam` via [MDAD](https://github.com/spantaleev/matrix-docker-ansible-deploy).
various bridges via [MDAD](https://github.com/spantaleev/matrix-docker-ansible-deploy).
```yaml
caches:
cache_autotuning:
max_cache_memory_usage: 2096103424
min_cache_ttl: 30s
target_cache_memory_usage: 1048051712
cache_entry_ttl: 1080m
expire_caches: true
global_factor: 2
per_cache_factors:
get_users_in_room: 5.0
sync_response_cache_duration: 2m
enable_metrics: true
experimental_features:
msc2409_to_device_messages_enabled: true
msc3202_device_masquerading: true
msc3202_transaction_extensions: true
msc4143_enabled: true
msc4222_enabled: true
msc4354_enabled: true
```
### Relevant log output
```shell
Aug 03 13:37:20 teahouse matrix-synapse-worker-federation-reader-0[2142028]: 2026-08-03 11:37:20,106 - synapse.replication.tcp.client - 383 - WARNING - PUT-552373 - Timed out waiting for repl stream 'typing' to reach 5497 (master); currently at: 5496
Aug 03 13:37:43 teahouse matrix-synapse-worker-federation-reader-0[2142028]: 2026-08-03 11:37:43,393 - synapse.replication.tcp.client - 383 - WARNING - PUT-552474 - Timed out waiting for repl stream 'device_lists' to reach 33308802 (master); currently at: 33308798
Aug 03 13:37:43 teahouse matrix-synapse-worker-federation-reader-0[2142028]: 2026-08-03 11:37:43,400 - synapse.replication.tcp.client - 383 - WARNING - PUT-552474 - Timed out waiting for repl stream 'device_lists' to reach 33308802 (master); currently at: 33308798
Aug 03 13:39:00 teahouse matrix-synapse-worker-federation-reader-0[2142028]: 2026-08-03 11:39:00,072 - synapse.crypto.keyring - 735 - WARNING - PerspectivesKeyFetcher-4873 - Key lookup failed from 'matrix.org': Not retrying server matrix.org because we tried it recently retry_last_ts=1785757115617 and we won't check for another retry_interval=600000ms.
Aug 03 13:41:26 teahouse matrix-synapse-worker-federation-reader-0[2142028]: 2026-08-03 11:41:26,173 - synapse.http.server - 879 - WARNING - PUT-553218 - Not sending response to request , already disconnected.
Aug 03 13:41:38 teahouse matrix-synapse-worker-federation-reader-0[2142028]: 2026-08-03 11:41:38,814 - synapse.http.server - 879 - WARNING - PUT-553238 - Not sending response to request , already disconnected.
Aug 03 13:41:39 teahouse matrix-synapse-worker-federation-reader-0[2142028]: 2026-08-03 11:41:38,816 - synapse.http.server - 879 - WARNING - PUT-553239 - Not sending response to request , already disconnected.
Aug 03 13:41:39 teahouse matrix-synapse-worker-federation-reader-0[2142028]: 2026-08-03 11:41:38,817 - synapse.http.server - 879 - WARNING - PUT-553237 - Not sending response to request , already disconnected.
Aug 03 13:41:39 teahouse matrix-synapse-worker-federation-reader-0[2142028]: 2026-08-03 11:41:39,055 - synapse.http.server - 879 - WARNING - PUT-553242 - Not sending response to request , already disconnected.
Aug 03 13:41:39 teahouse matrix-synapse-worker-federation-reader-0[2142028]: 2026-08-03 11:41:39,055 - synapse.http.server - 879 - WARNING - PUT-553235 - Not sending response to request , already disconnected.
Aug 03 13:41:39 teahouse matrix-synapse-worker-federation-reader-0[2142028]: 2026-08-03 11:41:39,056 - synapse.http.server - 879 - WARNING - PUT-553245 - Not sending response to request , already disconnected.
Aug 03 13:41:39 teahouse matrix-synapse-worker-federation-reader-0[2142028]: 2026-08-03 11:41:39,056 - synapse.http.server - 879 - WARNING - PUT-553166 - Not sending response to request , already disconnected.
Aug 03 13:41:39 teahouse matrix-synapse-worker-federation-reader-0[2142028]: 2026-08-03 11:41:39,057 - synapse.http.server - 879 - WARNING - PUT-553244 - Not sending response to request , already disconnected.
Aug 03 13:41:41 teahouse matrix-synapse-worker-federation-reader-0[2142028]: 2026-08-03 11:41:41,640 - synapse.http.server - 879 - WARNING - PUT-553247 - Not sending response to request , already disconnected.
Aug 03 13:41:41 teahouse matrix-synapse-worker-federation-reader-0[2142028]: 2026-08-03 11:41:41,645 - synapse.http.server - 879 - WARNING - PUT-553212 - Not sending response to request , already disconnected.
...
```
### Anything else that would be useful to know?
Running [`matrix-docker-ansible-deploy`](https://github.com/spantaleev/matrix-docker-ansible-deploy)
this is a regression from `1.156`, or at least I first experienced it in `1.157.1`
Contributor guide
Research direction
Start at the ReplicationFederationSendEduRestServlet and federation /_matrix/federation/v1/send entry points, comparing behavior between Synapse 1.156 and 1.157. Reproduce the laptop sleep/wake sequence and inspect the replication timeout and disconnected-request logs; done means the request backlog and DB saturation no longer persist after the client reconnects.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- postgresql, python
- Domain
- backend, databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100