Consider if "502 Bad Gateway" is the right status code for unavailable federated servers

This issue has been migrated from [#5837](https://github.com/matrix-org/synapse/issues/5837).

---

**Description:**

We have seen a number of loadbalancers that consider server failures like 502's to indicate an issue with the server as a whole, and not be scoped to just that specific URL on the server.

This has empirically lead to flip-flopping behaviour where a number of failing requests to a functioning homeserver that is attempting to contact a non-functioning homeserver over federation has lead to the functioning homeserver itself being marked as unavailable, and so traffic has not been routed to it by a loadbalancer.

This is definitely an issue with the loadbalancers, rather than synapse, but they may be out of our control, or the control of our synapse administrators, so we may still wish to address this.

The issue names no files, tests, or entry points. Start by locating Synapse's federation request failure handling and the current 502 response path, then assess how an alternative status would affect load balancers; done means a decided status-code policy with corresponding tests or documentation.

Written by the indexing model from the issue text.