element-hq / element-hq/synapse

Synapse 1.157 regressed to completely block DB i/o with ReplicationFederationSendEduRestServlet operations

Open
#20,037 3 comments 0 reactions 0 assignees View on GitHub
A-Federation A-Workers
Dominant language
Python
Stars
4.6k
Forks
600
Avg merge
5d 22h
Merged PRs (30d)
51

Description

### Description

Upon opening my laptop, Synapse `ReplicationFederationSendEduRestServlet` on my `stream_writer` worker (via [MDAD](https://github.com/spantaleev/matrix-docker-ansible-deploy)) goes crazy, hogs 1500%+ "DB usage" while still accumulating requests.
somehow the federation reader worker also seems involved.

FederationSendServlet and ReplicationFederationSendEduRestServlet increased traffic in the 'Request Count by arrival time' Grafana graph

ReplicationFederationSendEduRestServlet spiking in the 'Requests in flight' Grafana graph

### Steps to reproduce

- open nheko on laptop
- close laptop
- next morning, open laptop

after killing nheko, it takes my server ~4h to recover on its own, as depicted in the graph above. the number of requests continues to rise.

workaround is to restart synapse.

### Homeserver

selfhosted

### Synapse Version

1.157.2

### Installation Method

Other (please mention below)

### Database

single psql (Postgres). ported from original SQLite 5 or so years ago.

### Workers

Multiple workers

### Platform

Official Docker image in docker on Debian on VPS

### Configuration

tried disabling presence, but no luck. using one presence worker via [MDAD](https://github.com/spantaleev/matrix-docker-ansible-deploy).

using `synapse_http_antispam` via [MDAD](https://github.com/spantaleev/matrix-docker-ansible-deploy).

various bridges via [MDAD](https://github.com/spantaleev/matrix-docker-ansible-deploy).

```yaml
caches:
cache_autotuning:
max_cache_memory_usage: 2096103424
min_cache_ttl: 30s
target_cache_memory_usage: 1048051712
cache_entry_ttl: 1080m
expire_caches: true
global_factor: 2
per_cache_factors:
get_users_in_room: 5.0
sync_response_cache_duration: 2m
enable_metrics: true
experimental_features:
msc2409_to_device_messages_enabled: true
msc3202_device_masquerading: true
msc3202_transaction_extensions: true
msc4143_enabled: true
msc4222_enabled: true
msc4354_enabled: true
```

### Relevant log output

```shell
Aug 03 13:37:20 teahouse matrix-synapse-worker-federation-reader-0[2142028]: 2026-08-03 11:37:20,106 - synapse.replication.tcp.client - 383 - WARNING - PUT-552373 - Timed out waiting for repl stream 'typing' to reach 5497 (master); currently at: 5496
Aug 03 13:37:43 teahouse matrix-synapse-worker-federation-reader-0[2142028]: 2026-08-03 11:37:43,393 - synapse.replication.tcp.client - 383 - WARNING - PUT-552474 - Timed out waiting for repl stream 'device_lists' to reach 33308802 (master); currently at: 33308798
Aug 03 13:37:43 teahouse matrix-synapse-worker-federation-reader-0[2142028]: 2026-08-03 11:37:43,400 - synapse.replication.tcp.client - 383 - WARNING - PUT-552474 - Timed out waiting for repl stream 'device_lists' to reach 33308802 (master); currently at: 33308798
Aug 03 13:39:00 teahouse matrix-synapse-worker-federation-reader-0[2142028]: 2026-08-03 11:39:00,072 - synapse.crypto.keyring - 735 - WARNING - PerspectivesKeyFetcher-4873 - Key lookup failed from 'matrix.org': Not retrying server matrix.org because we tried it recently retry_last_ts=1785757115617 and we won't check for another retry_interval=600000ms.
Aug 03 13:41:26 teahouse matrix-synapse-worker-federation-reader-0[2142028]: 2026-08-03 11:41:26,173 - synapse.http.server - 879 - WARNING - PUT-553218 - Not sending response to request , already disconnected.
Aug 03 13:41:38 teahouse matrix-synapse-worker-federation-reader-0[2142028]: 2026-08-03 11:41:38,814 - synapse.http.server - 879 - WARNING - PUT-553238 - Not sending response to request , already disconnected.
Aug 03 13:41:39 teahouse matrix-synapse-worker-federation-reader-0[2142028]: 2026-08-03 11:41:38,816 - synapse.http.server - 879 - WARNING - PUT-553239 - Not sending response to request , already disconnected.
Aug 03 13:41:39 teahouse matrix-synapse-worker-federation-reader-0[2142028]: 2026-08-03 11:41:38,817 - synapse.http.server - 879 - WARNING - PUT-553237 - Not sending response to request , already disconnected.
Aug 03 13:41:39 teahouse matrix-synapse-worker-federation-reader-0[2142028]: 2026-08-03 11:41:39,055 - synapse.http.server - 879 - WARNING - PUT-553242 - Not sending response to request , already disconnected.
Aug 03 13:41:39 teahouse matrix-synapse-worker-federation-reader-0[2142028]: 2026-08-03 11:41:39,055 - synapse.http.server - 879 - WARNING - PUT-553235 - Not sending response to request , already disconnected.
Aug 03 13:41:39 teahouse matrix-synapse-worker-federation-reader-0[2142028]: 2026-08-03 11:41:39,056 - synapse.http.server - 879 - WARNING - PUT-553245 - Not sending response to request , already disconnected.
Aug 03 13:41:39 teahouse matrix-synapse-worker-federation-reader-0[2142028]: 2026-08-03 11:41:39,056 - synapse.http.server - 879 - WARNING - PUT-553166 - Not sending response to request , already disconnected.
Aug 03 13:41:39 teahouse matrix-synapse-worker-federation-reader-0[2142028]: 2026-08-03 11:41:39,057 - synapse.http.server - 879 - WARNING - PUT-553244 - Not sending response to request , already disconnected.
Aug 03 13:41:41 teahouse matrix-synapse-worker-federation-reader-0[2142028]: 2026-08-03 11:41:41,640 - synapse.http.server - 879 - WARNING - PUT-553247 - Not sending response to request , already disconnected.
Aug 03 13:41:41 teahouse matrix-synapse-worker-federation-reader-0[2142028]: 2026-08-03 11:41:41,645 - synapse.http.server - 879 - WARNING - PUT-553212 - Not sending response to request , already disconnected.
...
```

### Anything else that would be useful to know?

Running [`matrix-docker-ansible-deploy`](https://github.com/spantaleev/matrix-docker-ansible-deploy)

this is a regression from `1.156`, or at least I first experienced it in `1.157.1`

Contributor guide

Open the contributing guide

Research direction

Start at the ReplicationFederationSendEduRestServlet and federation /_matrix/federation/v1/send entry points, comparing behavior between Synapse 1.156 and 1.157. Reproduce the laptop sleep/wake sequence and inspect the replication timeout and disconnected-request logs; done means the request backlog and DB saturation no longer persist after the client reconnects.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, python
Domain
backend, databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.