Federation catch-up is not necessarily correct due to the sharding of the event stream

This issue has been migrated from [#15260](https://github.com/matrix-org/synapse/issues/15260).

---

In the `destinations` table, we have `last_successful_stream_ordering` which contains the `stream_ordering` of the most recent PDU that was successfully sent to that destination.

However, given that we have sharded the event stream (since federation catch-up was implemented, I believe), I think there is now a small race condition when the PDUs are not produced in `stream_ordering` order but yet all get enqueued for transmission to the same destination.

The race condition means that we might track the fact that we successfully transmitted PDUs up to stream ordering `x` when in fact the PDU at `x - 1` (etc) was not transmitted — and won't be transmitted during catch-up later.