element-hq / element-hq/synapse

`process_new_pulled_events_with_failed_pull_attempts` flooding requests to `beta2.matrix.org`, OOMing server, using all CPU

Open
#19,462 6 comments 0 reactions 0 assignees View on GitHub
A-Federation
Dominant language
Python
Stars
4.6k
Forks
600
Avg merge
5d 22h
Merged PRs (30d)
51

Description

### Description

Thousands of requests (over 350 per second) are being made to `POST matrix-federation://beta2.matrix.org/_matrix/policy/unstable/org.matrix.msc4284/event/$eventID/check`, and it is causing Synapse to run out of CPU and repeatedly get killed by the OOM reaper.

### Steps to reproduce

This issue started unprompted. Please observe a snippet of the logs attached.

### Homeserver

starstruck.systems

### Synapse Version

1.147.0

### Installation Method

pip (from PyPI)

### Database

postgres, mono, clean

### Workers

Multiple workers

### Platform

Ubuntu LXC, master process and worker processes controlled by systemd, same-metal postgres in another lxc, redis on the same lxc hosting synapse

### Configuration

Many experimental features enabled, presence disabled, synapse-http-antispam module enabled - full config at https://starstruck.systems/~star/synapse.yaml

### Relevant log output

```shell
Feb 13 00:02:37 starstruck synapse-generic-worker-2[25124]: 2026-02-13 00:02:37,268 - synapse.http.matrixfederationclient - 371 - INFO - _process_new_pulled_events_with_failed_pull_attempts-0-$ZVLtDGJZIaYO06omzkgjkiEyegdS8dBbSsKQVBmUXZo-$0vQtcZPah-77INQsPsuoWwfxC3MqiO5_BNGGnmHHNv8 - {POST-O-7097} [beta2.matrix.org] Completed request: 200 OK in 5.44 secs, got 24 bytes - POST matrix-federation://beta2.matrix.org/_matrix/policy/unstable/org.matrix.msc4284/event/$3U4ojmkgM05ta_mrIWglfAF5vGujKfEe6vYX5W-Tjtg/check
Feb 13 00:02:37 starstruck synapse-generic-worker-2[25124]: 2026-02-13 00:02:37,269 - synapse.http.matrixfederationclient - 371 - INFO - _process_new_pulled_events_with_failed_pull_attempts-0-$ZVLtDGJZIaYO06omzkgjkiEyegdS8dBbSsKQVBmUXZo-$0vQtcZPah-77INQsPsuoWwfxC3MqiO5_BNGGnmHHNv8 - {POST-O-7098} [beta2.matrix.org] Completed request: 200 OK in 5.44 secs, got 24 bytes - POST matrix-federation://beta2.matrix.org/_matrix/policy/unstable/org.matrix.msc4284/event/$JMsnCsRk6HRrUmHpo5bPhoYekXwztfGu4Csgwxrwn_g/check
Feb 13 00:02:37 starstruck synapse-generic-worker-2[25124]: 2026-02-13 00:02:37,269 - synapse.http.matrixfederationclient - 371 - INFO - _process_new_pulled_events_with_failed_pull_attempts-0-$ZVLtDGJZIaYO06omzkgjkiEyegdS8dBbSsKQVBmUXZo-$0vQtcZPah-77INQsPsuoWwfxC3MqiO5_BNGGnmHHNv8 - {POST-O-7099} [beta2.matrix.org] Completed request: 200 OK in 5.43 secs, got 24 bytes - POST matrix-federation://beta2.matrix.org/_matrix/policy/unstable/org.matrix.msc4284/event/$XJDVSk1ArMTjccbcAKMvSVn3YiOai4geH-3ImON83HM/check
Feb 13 00:02:37 starstruck synapse-generic-worker-2[25124]: 2026-02-13 00:02:37,269 - synapse.http.matrixfederationclient - 371 - INFO - _process_new_pulled_events_with_failed_pull_attempts-0-$ZVLtDGJZIaYO06omzkgjkiEyegdS8dBbSsKQVBmUXZo-$0vQtcZPah-77INQsPsuoWwfxC3MqiO5_BNGGnmHHNv8 - {POST-O-7100} [beta2.matrix.org] Completed request: 200 OK in 5.43 secs, got 24 bytes - POST matrix-federation://beta2.matrix.org/_matrix/policy/unstable/org.matrix.msc4284/event/$wAFj8pP5UNQFWmsbeQlEeUfSqYhqkEl0l2CrISL-PYg/check
Feb 13 00:02:37 starstruck synapse-generic-worker-2[25124]: 2026-02-13 00:02:37,270 - synapse.http.matrixfederationclient - 371 - INFO - _process_new_pulled_events_with_failed_pull_attempts-0-$ZVLtDGJZIaYO06omzkgjkiEyegdS8dBbSsKQVBmUXZo-$0vQtcZPah-77INQsPsuoWwfxC3MqiO5_BNGGnmHHNv8 - {POST-O-7101} [beta2.matrix.org] Completed request: 200 OK in 5.43 secs, got 24 bytes - POST matrix-federation://beta2.matrix.org/_matrix/policy/unstable/org.matrix.msc4284/event/$S1Ku_r-j_f8YKiiKWWeU4EgV-xlp6sJUDTS_XAeyW50/check
Feb 13 00:02:37 starstruck synapse-generic-worker-2[25124]: 2026-02-13 00:02:37,271 - synapse.http.matrixfederationclient - 838 - INFO - _process_new_pulled_events_with_failed_pull_attempts-0-$ZVLtDGJZIaYO06omzkgjkiEyegdS8dBbSsKQVBmUXZo-$0vQtcZPah-77INQsPsuoWwfxC3MqiO5_BNGGnmHHNv8 - {POST-O-278} [beta2.matrix.org] Request failed: POST matrix-federation://beta2.matrix.org/_matrix/policy/unstable/org.matrix.msc4284/event/$V2V9EkUCAWw6yc2931QfESP5FaFIF32tsisUnuB6_-E/check: ResponseNeverReceived:[ConnectionLost('Connection lost')]
Feb 13 00:02:37 starstruck synapse-generic-worker-2[25124]: 2026-02-13 00:02:37,271 - synapse.http.matrixfederationclient - 838 - INFO - _process_new_pulled_events_with_failed_pull_attempts-0-$ZVLtDGJZIaYO06omzkgjkiEyegdS8dBbSsKQVBmUXZo-$0vQtcZPah-77INQsPsuoWwfxC3MqiO5_BNGGnmHHNv8 - {POST-O-338} [beta2.matrix.org] Request failed: POST matrix-federation://beta2.matrix.org/_matrix/policy/unstable/org.matrix.msc4284/event/$PgPeF2xMR0WdCki1kNZ8QA1iWp527OSOY338lcIWqbM/check: RequestTransmissionFailed:[ConnectionLost('Connection lost')]
Feb 13 00:02:37 starstruck synapse-generic-worker-2[25124]: 2026-02-13 00:02:37,328 - synapse.http.matrixfederationclient - 838 - INFO - _process_new_pulled_events_with_failed_pull_attempts-0-$ZVLtDGJZIaYO06omzkgjkiEyegdS8dBbSsKQVBmUXZo-$0vQtcZPah-77INQsPsuoWwfxC3MqiO5_BNGGnmHHNv8 - {POST-O-220} [beta2.matrix.org] Request failed: POST matrix-federation://beta2.matrix.org/_matrix/policy/unstable/org.matrix.msc4284/event/$-cRs_m-7QHvpVfykMBeJFHW2PsbQ7HqgLllALyRT0tQ/check: ResponseNeverReceived:[ConnectionLost('Connection lost')]
Feb 13 00:02:37 starstruck synapse-generic-worker-2[25124]: 2026-02-13 00:02:37,328 - synapse.http.matrixfederationclient - 838 - INFO - _process_new_pulled_events_with_failed_pull_attempts-0-$ZVLtDGJZIaYO06omzkgjkiEyegdS8dBbSsKQVBmUXZo-$0vQtcZPah-77INQsPsuoWwfxC3MqiO5_BNGGnmHHNv8 - {POST-O-247} [beta2.matrix.org] Request failed: POST matrix-federation://beta2.matrix.org/_matrix/policy/unstable/org.matrix.msc4284/event/$RQS60G936rILBNMf4yJuaNhFJ87xL8krXYJO-VBbpf0/check: RequestTransmissionFailed:[ConnectionLost('Connection lost')]
Feb 13 00:02:37 starstruck synapse-generic-worker-2[25124]: 2026-02-13 00:02:37,328 - synapse.http.matrixfederationclient - 838 - INFO - _process_new_pulled_events_with_failed_pull_attempts-0-$ZVLtDGJZIaYO06omzkgjkiEyegdS8dBbSsKQVBmUXZo-$0vQtcZPah-77INQsPsuoWwfxC3MqiO5_BNGGnmHHNv8 - {POST-O-240} [beta2.matrix.org] Request failed: POST matrix-federation://beta2.matrix.org/_matrix/policy/unstable/org.matrix.msc4284/event/$K6nKZ5cWsY5FcEnmRrQhFb6LSnXfpp7U749Hjjdv2N8/check: RequestTransmissionFailed:[ConnectionLost('Connection lost')]
Feb 13 00:02:37 starstruck synapse-generic-worker-2[25124]: 2026-02-13 00:02:37,329 - synapse.http.matrixfederationclient - 838 - INFO - _process_new_pulled_events_with_failed_pull_attempts-0-$ZVLtDGJZIaYO06omzkgjkiEyegdS8dBbSsKQVBmUXZo-$0vQtcZPah-77INQsPsuoWwfxC3MqiO5_BNGGnmHHNv8 - {POST-O-248} [beta2.matrix.org] Request failed: POST matrix-federation://beta2.matrix.org/_matrix/policy/unstable/org.matrix.msc4284/event/$2NGN5-Rup_TnLJybCre9-etAv9fxSsauEA3Cm_f2Tko/check: ResponseNeverReceived:[ConnectionLost('Connection lost')]
```

### Anything else that would be useful to know?

A two-minute log snippet: [logs.txt](https://github.com/user-attachments/files/25277666/logs.txt)
The full hour from the issue starting to the issue being noticed (300MB unzipped): [logs.txt.gz](https://github.com/user-attachments/files/25277670/logs.txt.gz)

Image

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.