element-hq / element-hq/synapse
[metrics] Background process 'delayed_events.notify_new_event' threw an exception
- Dominant language
- Python
- Stars
- 4.6k
- Forks
- 600
- Avg merge
- 5d 22h
- Merged PRs (30d)
- 51
Description
### Description
Metrics keeps throwing errors from delayed_events for rejected events
`synapse.api.errors.NotFoundError: 404: Could not find event $ByeSbu0iUhv0RB4MSH047mwIObVRuHLYspnLZDjklOk`
From `events` table:
|topological_ordering|event_id|type|room_id|content|unrecognized_keys|processed|outlier|depth|origin_server_ts|received_ts|sender|contains_url|instance_name|stream_ordering|state_key|rejection_reason|
|--------------------|--------|----|-------|-------|-----------------|---------|-------|-----|----------------|-----------|------|------------|-------------|---------------|---------|----------------|
|17782|$ByeSbu0iUhv0RB4MSH047mwIObVRuHLYspnLZDjklOk|m.room.member|!NasysSDfxKxZBzJJoE:matrix.org|||true|true|17782|1611758224059|1745096815930|@yan:yetanothernerd.xyz|false|event_stream_writer-0|-43538330|@yan:yetanothernerd.xyz|auth_error|
### Steps to reproduce
- join !NasysSDfxKxZBzJJoE:matrix.org?via=matrix.org
### Homeserver
rory.gay
### Synapse Version
1.127.1
### Installation Method
Other (please mention below)
### Database
single postgres, no port/restore
### Workers
Multiple workers
### Platform
nixos/unstable, dedicated host
### Configuration
presence enabled
experimental_features = {
"msc2815_enabled" = true; # Redacted event content
"msc3026_enabled" = true; # Busy presence
"msc3266_enabled" = true; # Room summary API
"msc3916_authenticated_media_enabled" = true; # Authenticated media
"msc3823_account_suspension" = true; # Account suspension
"msc4151_enabled" = true; # Report room API (CS-API)
};
https://cgit.rory.gay/Rory-Open-Architecture.git/tree/host/Rory-ovh/services/matrix/synapse/synapse-main.nix
### Relevant log output
```shell
Apr 19 22:09:01 Rory-nginx synapse[2570433]: synapse.metrics.background_process_metrics: [delayed_events.notify_new_event-7264596] Background process 'delayed_events.notify_new_event' threw an exception
Traceback (most recent call last):
File "/nix/store/ibc5r4abj06040pjsaa8qlc8s9jl1v00-matrix-synapse-1.127.1/lib/python3.12/site-packages/synapse/metrics/background_process_metrics.py", line 251, in run
return await func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/nix/store/ibc5r4abj06040pjsaa8qlc8s9jl1v00-matrix-synapse-1.127.1/lib/python3.12/site-packages/synapse/handlers/delayed_events.py", line 138, in process
await self._unsafe_process_new_event()
File "/nix/store/ibc5r4abj06040pjsaa8qlc8s9jl1v00-matrix-synapse-1.127.1/lib/python3.12/site-packages/synapse/handlers/delayed_events.py", line 183, in _unsafe_process_new_event
await self._handle_state_deltas(deltas)
File "/nix/store/ibc5r4abj06040pjsaa8qlc8s9jl1v00-matrix-synapse-1.127.1/lib/python3.12/site-packages/synapse/handlers/delayed_events.py", line 210, in _handle_state_deltas
event = await self._store.get_event(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/nix/store/ibc5r4abj06040pjsaa8qlc8s9jl1v00-matrix-synapse-1.127.1/lib/python3.12/site-packages/synapse/storage/databases/main/events_worker.py", line 548, in get_event
raise NotFoundError("Could not find event %s" % (event_id,))
synapse.api.errors.NotFoundError: 404: Could not find event $ByeSbu0iUhv0RB4MSH047mwIObVRuHLYspnLZDjklOk
```
### Anything else that would be useful to know?
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.