element-hq / element-hq/synapse

federation_sender repeatedly wakes up destinations ('Destination example.org has outstanding catch-up, waking up') making no progress

Open
#18,500 0 comments 0 reactions 0 assignees View on GitHub
A-Federation O-Occasional S-Minor T-Defect
Dominant language
Python
Stars
4.6k
Forks
600
Avg merge
5d 22h
Merged PRs (30d)
51

Description

### Description

If I am in a room and then leave that room,
it's possible to enter a state where some the federation sender will perpetually wake up that destination to send it new events for that room,
without making any progress since we are no longer in that room.

(reporting on behalf of others + matrix.org)

This logic selects destinations to wake up:
```sql
WITH pdu_destinations AS (
SELECT DISTINCT destination FROM destination_rooms
LEFT JOIN destinations USING (destination)
WHERE
destination > ?
AND destination_rooms.stream_ordering > COALESCE(destinations.last_successful_stream_ordering, 0)
AND (
destinations.retry_last_ts IS NULL OR
destinations.retry_last_ts + destinations.retry_interval < ?
)
ORDER BY destination
LIMIT 25
), to_device_destinations AS (
SELECT DISTINCT destination FROM device_federation_outbox
LEFT JOIN destinations USING (destination)
WHERE
destination > ?
AND (
destinations.retry_last_ts IS NULL OR
destinations.retry_last_ts + destinations.retry_interval < ?
)
ORDER BY destination
LIMIT 25
)

SELECT destination FROM pdu_destinations
UNION SELECT destination FROM to_device_destinations
ORDER BY destination
LIMIT 25
```

so it's enough to have `destination_rooms` rows for a room you're not in, to always have a destination to wake up that you can't make progress with.

### Steps to reproduce

untested

- join federated room
- send events, with some servers not online (so it enters catch-up mode) — those servers should not be in another room that you're in
- leave room
- do not purge the room
- view logs

### Homeserver

matrix.org, others

### Synapse Version

1.130.0

### Installation Method

Docker (matrixdotorg/synapse)

### Database

Postgres

### Workers

Multiple workers

### Platform

-

### Configuration

_No response_

### Relevant log output

```shell
part of `wake_destinations_needing_catchup-0`
```

### Anything else that would be useful to know?

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.