element-hq / element-hq/synapse
federation_sender repeatedly wakes up destinations ('Destination example.org has outstanding catch-up, waking up') making no progress
- Dominant language
- Python
- Stars
- 4.6k
- Forks
- 600
- Avg merge
- 5d 22h
- Merged PRs (30d)
- 51
Description
### Description
If I am in a room and then leave that room,
it's possible to enter a state where some the federation sender will perpetually wake up that destination to send it new events for that room,
without making any progress since we are no longer in that room.
(reporting on behalf of others + matrix.org)
This logic selects destinations to wake up:
```sql
WITH pdu_destinations AS (
SELECT DISTINCT destination FROM destination_rooms
LEFT JOIN destinations USING (destination)
WHERE
destination > ?
AND destination_rooms.stream_ordering > COALESCE(destinations.last_successful_stream_ordering, 0)
AND (
destinations.retry_last_ts IS NULL OR
destinations.retry_last_ts + destinations.retry_interval < ?
)
ORDER BY destination
LIMIT 25
), to_device_destinations AS (
SELECT DISTINCT destination FROM device_federation_outbox
LEFT JOIN destinations USING (destination)
WHERE
destination > ?
AND (
destinations.retry_last_ts IS NULL OR
destinations.retry_last_ts + destinations.retry_interval < ?
)
ORDER BY destination
LIMIT 25
)
SELECT destination FROM pdu_destinations
UNION SELECT destination FROM to_device_destinations
ORDER BY destination
LIMIT 25
```
so it's enough to have `destination_rooms` rows for a room you're not in, to always have a destination to wake up that you can't make progress with.
### Steps to reproduce
untested
- join federated room
- send events, with some servers not online (so it enters catch-up mode) — those servers should not be in another room that you're in
- leave room
- do not purge the room
- view logs
### Homeserver
matrix.org, others
### Synapse Version
1.130.0
### Installation Method
Docker (matrixdotorg/synapse)
### Database
Postgres
### Workers
Multiple workers
### Platform
-
### Configuration
_No response_
### Relevant log output
```shell
part of `wake_destinations_needing_catchup-0`
```
### Anything else that would be useful to know?
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.