An idle event persister can hold up outbound federation of PDUs and other processes

This issue has been migrated from [#15595](https://github.com/matrix-org/synapse/issues/15595).

---

During a recent federation outage on matrix.org, where federation senders would get stuck for minutes, @\richvdh noticed that the minimum stream position of `RoomStreamToken`s was stuck because event_persister-4 wasn't advancing.

https://matrix.to/#/!yHWhpxlXVaLcsgDUKb:matrix.org/$mve7Xf2PoCru5c6wrcWsNUXQEf_4rtqXRv3Gbyi6E8M?via=matrix.org&via=element.io&via=vector.modular.im

---

When an event persister is idle, the minimum stream position in `RoomStreamToken`s will be stuck at the last persisted stream position of the idle event persister plus any continuous run of stream positions seen over replication after that. That is, the minimum stream position gets stuck at the first gap. See [here](https://github.com/matrix-org/synapse/blob/36df9c5e36cbad2a378d922085453726a21ae80c/synapse/storage/util/id_generators.py#L759-L794) for how the minimum stream position is calculated.

For an explanation of the fields in a RoomStreamToken, see
https://github.com/matrix-org/synapse/blob/36df9c5e36cbad2a378d922085453726a21ae80c/synapse/types/__init__.py#L476-L496

The federation senders use this minimum stream position to determine where it is safe to process up to (since new events can't appear with an earlier stream position). Thus when the minimum stream position gets stuck, the federation senders stop making progress even when there are new events from local users needing to be sent.

https://github.com/matrix-org/synapse/blob/3bf973edc7ecf911daab6d8c58d1264891f3ed39/synapse/replication/tcp/client.py#L176-L188
https://github.com/matrix-org/synapse/blob/36df9c5e36cbad2a378d922085453726a21ae80c/synapse/federation/sender/__init__.py#L444-L453
https://github.com/matrix-org/synapse/blob/36df9c5e36cbad2a378d922085453726a21ae80c/synapse/federation/sender/__init__.py#L467-L473

---

The idle event persister could likely do something to fix the problem, since it can tell when it is behind.
https://github.com/matrix-org/synapse/blob/36df9c5e36cbad2a378d922085453726a21ae80c/synapse/storage/util/id_generators.py#L759-L794
Note that `_persisted_upto_position` can end up ahead of the event persister's own position in `_current_positions` if it has nothing in flight. However, the event persister's own position doesn't appear to be updated and `_persisted_upto_position` isn't broadcast over replication.

Even if we did broadcast `_persisted_upto_position` over replication, this would only work for a single idle event persister. When there are two or more idle event persisters, we would just get stuck.

Note that the second half of the code is responsible for advancing the minimum stream position up to the first gap in stream positions.

Start in synapse/storage/util/id_generators.py at the minimum stream-position calculation and inspect how idle event persisters update _current_positions and _persisted_upto_position. Then trace the referenced replication client and federation sender paths. Done means idle persisters no longer leave a gap that prevents the minimum RoomStreamToken from advancing and outbound federation makes progress.

Written by the indexing model from the issue text.