Events sent immediately after joining can be incorrectly soft-failed

This issue has been migrated from [#10066](https://github.com/matrix-org/synapse/issues/10066).

---

### Description

The Gitter bridge seems to have reproduced a race condition in federation where because the appservice joins and sends a message in quick succession, the message can out run the `join` which causes the message to `soft_fail`.

To see an actual reproduction, there is a missing message in the `!SrkiFczPSWmYrlSNYF:matrix.org` room for `matrix.org` homeserver users (`$mMJdjGPC2GaoWK3pLApa_O2vBoeuQwYtP6iY6XzbrfY`). This is tracked on Gitter by https://gitlab.com/gitterHQ/webapp/-/issues/2770#note_583925093

The event exists in the `gitter.im` homeserver database and is visible to the `gitter.im` homeserver where the Gitter bridge appservice operates. The message is also visible to any new homeserver that comes in the room and backfills the messages. It's only not visible on the `matrix.org` homeserver.


The message also exists on the `matrix.org` homeserver database but is `soft_failed` (thanks to @richvdh for [checking the database](https://matrix.to/#/!oBWlVutBAdQYcYUsAc:sw1v.org/$6HDZd4Iv9pSZwaSXRW8DIF6Sxtv1AhlQnrlzFMHCFK0?via=jki.re&via=matrix.org&via=vector.modular.im))

@Half-Shot has also seen this happen a few times with the IRC bridge.

As @leonerd from https://github.com/matrix-org/synapse/issues/1444 and @Half-Shot mentioned, this is probably not a problem for normal users because the time taken to join and then send a message is some number of seconds. Whereas with appservices, can join and send an event in quick succession (almost instantly).

Related issues:

- https://github.com/matrix-org/synapse/issues/6536: events sent immediately after joining can be incorrectly soft-failed
- https://github.com/matrix-org/synapse/issues/7744: events get soft-failed when the `federation_inbound` worker is busy
- https://github.com/matrix-org/synapse/issues/1444: Reproduces with a `join`, set profile, `membership` event (not over federation though).
- https://github.com/matrix-org/synapse/issues/9329

### Steps to reproduce

1. Create a room on HS1, `!my-room:hs1`
1. Setup HS2 with an appservice
1. Join and send a message through the appservice to `!my-room:hs1`
- The appservice is probably not necessary to reproduce. Just send a `join` and message event from HS2
1. Check whether the message is visible on hs1
1. Since it's a race condition, this probably does not always reproduce

### Version information

- **Homeserver**: `gitter.im` -> `matrix.org`