Synapse is sending events to appservices multiple times

This issue has been migrated from [#11447](https://github.com/matrix-org/synapse/issues/11447).

---

### Description

As described in the issue https://github.com/matrix-org/matrix-appservice-irc/issues/1499 synapse is sending events to appservices multiple times when e.g. the federation is temporarily broken. This ends up creating duplicate messages. We've seen as much as 4x the same conversation snippet being posted.

During a discussion in https://matrix.to/#/#irc:matrix.org it was pointed out ([message](https://matrix.to/#/!BAXLHOFjvDKUeLafmO:matrix.org/$ip/ByE3DsKaoNcr12chVhmzsZ5+ouiFbYadW8HS41HA?via=matrix.org&via=libera.chat&via=mozilla.org)) that this is a spec violation.

Quote of the relevant part (https://spec.matrix.org/v1.1/application-service-api/#pushing-events):

> The events sent to the application service should be linearised, as if they were from the event stream. The homeserver MUST maintain a queue of transactions to send to the application service. If the application service cannot be reached, the homeserver SHOULD backoff exponentially until the application service is reachable again. As application services cannot modify the events in any way, these requests can be made without blocking other aspects of the homeserver. Homeservers MUST NOT alter (e.g. add more) events they were going to send within that transaction ID on retries, as the application service may have already processed the events.

Especially the last part (`as the application service may have already processed the events.`) is relevant in this context.

I can provide logs of the appservice & synapse in the given time frames. Contact me on matrix `@andi:kack.it`.

### Steps to reproduce

- See steps in the linked IRC appservice issue

### Version information

- **Homeserver**:

If not matrix.org:

- **Version**: 1.47.1

- **Install method**:

- **Platform**: NixOS