element-hq / element-hq/synapse
`process_new_pulled_events_with_failed_pull_attempts` flooding requests to `beta2.matrix.org`, OOMing server, using all CPU
- Dominant language
- Python
- Stars
- 4.6k
- Forks
- 600
- Avg merge
- 5d 22h
- Merged PRs (30d)
- 51
Description
### Description
Thousands of requests (over 350 per second) are being made to `POST matrix-federation://beta2.matrix.org/_matrix/policy/unstable/org.matrix.msc4284/event/$eventID/check`, and it is causing Synapse to run out of CPU and repeatedly get killed by the OOM reaper.
### Steps to reproduce
This issue started unprompted. Please observe a snippet of the logs attached.
### Homeserver
starstruck.systems
### Synapse Version
1.147.0
### Installation Method
pip (from PyPI)
### Database
postgres, mono, clean
### Workers
Multiple workers
### Platform
Ubuntu LXC, master process and worker processes controlled by systemd, same-metal postgres in another lxc, redis on the same lxc hosting synapse
### Configuration
Many experimental features enabled, presence disabled, synapse-http-antispam module enabled - full config at https://starstruck.systems/~star/synapse.yaml
### Relevant log output
```shell
Feb 13 00:02:37 starstruck synapse-generic-worker-2[25124]: 2026-02-13 00:02:37,268 - synapse.http.matrixfederationclient - 371 - INFO - _process_new_pulled_events_with_failed_pull_attempts-0-$ZVLtDGJZIaYO06omzkgjkiEyegdS8dBbSsKQVBmUXZo-$0vQtcZPah-77INQsPsuoWwfxC3MqiO5_BNGGnmHHNv8 - {POST-O-7097} [beta2.matrix.org] Completed request: 200 OK in 5.44 secs, got 24 bytes - POST matrix-federation://beta2.matrix.org/_matrix/policy/unstable/org.matrix.msc4284/event/$3U4ojmkgM05ta_mrIWglfAF5vGujKfEe6vYX5W-Tjtg/check
Feb 13 00:02:37 starstruck synapse-generic-worker-2[25124]: 2026-02-13 00:02:37,269 - synapse.http.matrixfederationclient - 371 - INFO - _process_new_pulled_events_with_failed_pull_attempts-0-$ZVLtDGJZIaYO06omzkgjkiEyegdS8dBbSsKQVBmUXZo-$0vQtcZPah-77INQsPsuoWwfxC3MqiO5_BNGGnmHHNv8 - {POST-O-7098} [beta2.matrix.org] Completed request: 200 OK in 5.44 secs, got 24 bytes - POST matrix-federation://beta2.matrix.org/_matrix/policy/unstable/org.matrix.msc4284/event/$JMsnCsRk6HRrUmHpo5bPhoYekXwztfGu4Csgwxrwn_g/check
Feb 13 00:02:37 starstruck synapse-generic-worker-2[25124]: 2026-02-13 00:02:37,269 - synapse.http.matrixfederationclient - 371 - INFO - _process_new_pulled_events_with_failed_pull_attempts-0-$ZVLtDGJZIaYO06omzkgjkiEyegdS8dBbSsKQVBmUXZo-$0vQtcZPah-77INQsPsuoWwfxC3MqiO5_BNGGnmHHNv8 - {POST-O-7099} [beta2.matrix.org] Completed request: 200 OK in 5.43 secs, got 24 bytes - POST matrix-federation://beta2.matrix.org/_matrix/policy/unstable/org.matrix.msc4284/event/$XJDVSk1ArMTjccbcAKMvSVn3YiOai4geH-3ImON83HM/check
Feb 13 00:02:37 starstruck synapse-generic-worker-2[25124]: 2026-02-13 00:02:37,269 - synapse.http.matrixfederationclient - 371 - INFO - _process_new_pulled_events_with_failed_pull_attempts-0-$ZVLtDGJZIaYO06omzkgjkiEyegdS8dBbSsKQVBmUXZo-$0vQtcZPah-77INQsPsuoWwfxC3MqiO5_BNGGnmHHNv8 - {POST-O-7100} [beta2.matrix.org] Completed request: 200 OK in 5.43 secs, got 24 bytes - POST matrix-federation://beta2.matrix.org/_matrix/policy/unstable/org.matrix.msc4284/event/$wAFj8pP5UNQFWmsbeQlEeUfSqYhqkEl0l2CrISL-PYg/check
Feb 13 00:02:37 starstruck synapse-generic-worker-2[25124]: 2026-02-13 00:02:37,270 - synapse.http.matrixfederationclient - 371 - INFO - _process_new_pulled_events_with_failed_pull_attempts-0-$ZVLtDGJZIaYO06omzkgjkiEyegdS8dBbSsKQVBmUXZo-$0vQtcZPah-77INQsPsuoWwfxC3MqiO5_BNGGnmHHNv8 - {POST-O-7101} [beta2.matrix.org] Completed request: 200 OK in 5.43 secs, got 24 bytes - POST matrix-federation://beta2.matrix.org/_matrix/policy/unstable/org.matrix.msc4284/event/$S1Ku_r-j_f8YKiiKWWeU4EgV-xlp6sJUDTS_XAeyW50/check
Feb 13 00:02:37 starstruck synapse-generic-worker-2[25124]: 2026-02-13 00:02:37,271 - synapse.http.matrixfederationclient - 838 - INFO - _process_new_pulled_events_with_failed_pull_attempts-0-$ZVLtDGJZIaYO06omzkgjkiEyegdS8dBbSsKQVBmUXZo-$0vQtcZPah-77INQsPsuoWwfxC3MqiO5_BNGGnmHHNv8 - {POST-O-278} [beta2.matrix.org] Request failed: POST matrix-federation://beta2.matrix.org/_matrix/policy/unstable/org.matrix.msc4284/event/$V2V9EkUCAWw6yc2931QfESP5FaFIF32tsisUnuB6_-E/check: ResponseNeverReceived:[ConnectionLost('Connection lost')]
Feb 13 00:02:37 starstruck synapse-generic-worker-2[25124]: 2026-02-13 00:02:37,271 - synapse.http.matrixfederationclient - 838 - INFO - _process_new_pulled_events_with_failed_pull_attempts-0-$ZVLtDGJZIaYO06omzkgjkiEyegdS8dBbSsKQVBmUXZo-$0vQtcZPah-77INQsPsuoWwfxC3MqiO5_BNGGnmHHNv8 - {POST-O-338} [beta2.matrix.org] Request failed: POST matrix-federation://beta2.matrix.org/_matrix/policy/unstable/org.matrix.msc4284/event/$PgPeF2xMR0WdCki1kNZ8QA1iWp527OSOY338lcIWqbM/check: RequestTransmissionFailed:[ConnectionLost('Connection lost')]
Feb 13 00:02:37 starstruck synapse-generic-worker-2[25124]: 2026-02-13 00:02:37,328 - synapse.http.matrixfederationclient - 838 - INFO - _process_new_pulled_events_with_failed_pull_attempts-0-$ZVLtDGJZIaYO06omzkgjkiEyegdS8dBbSsKQVBmUXZo-$0vQtcZPah-77INQsPsuoWwfxC3MqiO5_BNGGnmHHNv8 - {POST-O-220} [beta2.matrix.org] Request failed: POST matrix-federation://beta2.matrix.org/_matrix/policy/unstable/org.matrix.msc4284/event/$-cRs_m-7QHvpVfykMBeJFHW2PsbQ7HqgLllALyRT0tQ/check: ResponseNeverReceived:[ConnectionLost('Connection lost')]
Feb 13 00:02:37 starstruck synapse-generic-worker-2[25124]: 2026-02-13 00:02:37,328 - synapse.http.matrixfederationclient - 838 - INFO - _process_new_pulled_events_with_failed_pull_attempts-0-$ZVLtDGJZIaYO06omzkgjkiEyegdS8dBbSsKQVBmUXZo-$0vQtcZPah-77INQsPsuoWwfxC3MqiO5_BNGGnmHHNv8 - {POST-O-247} [beta2.matrix.org] Request failed: POST matrix-federation://beta2.matrix.org/_matrix/policy/unstable/org.matrix.msc4284/event/$RQS60G936rILBNMf4yJuaNhFJ87xL8krXYJO-VBbpf0/check: RequestTransmissionFailed:[ConnectionLost('Connection lost')]
Feb 13 00:02:37 starstruck synapse-generic-worker-2[25124]: 2026-02-13 00:02:37,328 - synapse.http.matrixfederationclient - 838 - INFO - _process_new_pulled_events_with_failed_pull_attempts-0-$ZVLtDGJZIaYO06omzkgjkiEyegdS8dBbSsKQVBmUXZo-$0vQtcZPah-77INQsPsuoWwfxC3MqiO5_BNGGnmHHNv8 - {POST-O-240} [beta2.matrix.org] Request failed: POST matrix-federation://beta2.matrix.org/_matrix/policy/unstable/org.matrix.msc4284/event/$K6nKZ5cWsY5FcEnmRrQhFb6LSnXfpp7U749Hjjdv2N8/check: RequestTransmissionFailed:[ConnectionLost('Connection lost')]
Feb 13 00:02:37 starstruck synapse-generic-worker-2[25124]: 2026-02-13 00:02:37,329 - synapse.http.matrixfederationclient - 838 - INFO - _process_new_pulled_events_with_failed_pull_attempts-0-$ZVLtDGJZIaYO06omzkgjkiEyegdS8dBbSsKQVBmUXZo-$0vQtcZPah-77INQsPsuoWwfxC3MqiO5_BNGGnmHHNv8 - {POST-O-248} [beta2.matrix.org] Request failed: POST matrix-federation://beta2.matrix.org/_matrix/policy/unstable/org.matrix.msc4284/event/$2NGN5-Rup_TnLJybCre9-etAv9fxSsauEA3Cm_f2Tko/check: ResponseNeverReceived:[ConnectionLost('Connection lost')]
```
### Anything else that would be useful to know?
A two-minute log snippet: [logs.txt](https://github.com/user-attachments/files/25277666/logs.txt)
The full hour from the issue starting to the issue being noticed (300MB unzipped): [logs.txt.gz](https://github.com/user-attachments/files/25277670/logs.txt.gz)
Contributor guide
Assessment
This issue has not been assessed yet.