`handle_incoming_transaction` goes on processing events in the background after returning an error

This issue has been migrated from [#8844](https://github.com/matrix-org/synapse/issues/8844).

---

In [`handle_incoming_transaction`](https://github.com/matrix-org/synapse/blob/950bb0305fc3c114f456eb4e2a806014150545d2/synapse/federation/federation_server.py#L203-L260)(2020-11-19), if there is a fatal error processing any of the events in a federation transaction (for example, if `check_server_matches_acl` raises an exception), then we will respond immediately to the `/send` request with an error, but will keep processing events for other rooms in the background.

We should wait for all the `process_pdus_for_room` calls to complete before bailing out, so that we can report the metrics properly.

## Relevant code (2021-11-10)

https://github.com/matrix-org/synapse/blob/6ce19b94e84eb6ad83ef303f88d8bd59a3d414e6/synapse/federation/federation_server.py#L446-L448

From some squinting and assumptions about the code, `concurrently_execute` will iterate over every item regardless if others errored out because of `consumeErrors`, but we also have `.addErrback(unwrapFirstError)` to throw the first error that occured.

Related docs: https://docs.twistedmatrix.com/en/twisted-18.9.0/core/howto/defer.html#gatherresults

https://github.com/matrix-org/synapse/blob/6ce19b94e84eb6ad83ef303f88d8bd59a3d414e6/synapse/util/async_helpers.py#L191-L210

Start in synapse/federation/federation_server.py at handle_incoming_transaction and the process_pdus_for_room calls. Read synapse/util/async_helpers.py lines 191-210 and the linked Twisted gatherResults documentation to understand the error-handling behavior. Done means all room-processing calls have completed before the error response is returned, so metrics are reported correctly.

Written by the indexing model from the issue text.