Implement prometheus metric that shows incoming federation lag (federation_reader delays in event receiving)

This issue has been migrated from [#10553](https://github.com/matrix-org/synapse/issues/10553).

---

**Description:**

At now [grafana/synapse.json](https://github.com/matrix-org/synapse/blob/master/contrib/grafana/synapse.json) shows "Age of last processed event" only for federation_sender and appservice_sender (outgoing federation). Is it possible to expose same metrics for federation_receiver, or something similar?

I have a problem, that my public ru-matrix.org homeserver [have a huge incoming federation delays](https://github.com/matrix-org/synapse/issues/10562) from most of popular homeservers (matrix.org, tchnics.de, etc) - hours, even for newly created DMs. Invites and first message are received momentually, but next messages have delays of several hours! Even starting 6 federation_reader workers didn't help (I had launched 6 workers for 2 days span and seen no improvements).

So, I want to understand where can be the source of that problem via some charts, related to lag in incoming federation.

Now for incoming federation analysis I see only "Incoming PDU/EDU rate" chart for this, and that's all?

Start with contrib/grafana/synapse.json and the federation_reader workers described in the issue; compare the existing federation_sender and appservice_sender age charts with the incoming-federation data already shown there. Done means an exposed incoming-federation lag metric and corresponding chart that makes receiving delays observable.

Written by the indexing model from the issue text.