Provide ways to analyze incoming federation delay issues and other problems with incoming message delays

This issue has been migrated from [#10562](https://github.com/matrix-org/synapse/issues/10562).

---

Our public homeserver ru-matrix.org with not so large userbase (less than 20 active users) have a big problem with incoming federation: messages from popular homeservers (eg matrix.org, tchncs.de) are delayed by several hours! And same thing with encryption keys. But from non-popular homeservers messages are delivered in several seconds.

And I, as admin of this server, can't even understand where can be the problem!

Server have enough free space, CPU is not so busy (8 CPU with average load less than 1), memory is not full too (16 gigabytes).
I see no errors in Synapse and PostgreSQL logs too. I have all prometheus metrics for Synapse, Linux, Postgres and see no problems in them too...

I even tried to start 6 separate workers for federation in 10 hours span - but this isn't help, delays stays at more than hour spans.

So, please, give here for me (and for other Synapse users that have similar problems) some information to analyze, where can be the problem, how to investigate it, and what can I try more to solve it?

> Please don't file github issues asking for support.

I already asked about this in [`#synapse:matrix.org`](https://matrix.to/#/#synapse:matrix.org), but not receive any useful advice, so filling long-standing issue to hope receive something useful :)