[Presence] Huge spike in CPU usage/Federation traffic approximately every 25 minutes

This issue has been migrated from [#15878](https://github.com/matrix-org/synapse/issues/15878).

---

There is a `FEDERATION_PING_INTERVAL` for when to send a ping over federation every 25 minutes that keeps a user from being marked 'offline' before the 30 minute `FEDERATION_TIMEOUT` hits.

https://github.com/element-hq/synapse/blob/4d8905a15a417ed0054ec2533d243932d890bbbd/synapse/handlers/presence.py#L194-L198


This appears to be the replication notifier system ramping up and queueing a bunch of federation sending requests over approximately 1 minute worth of time(give or take a few seconds)
Images




There is a database hit during this to `get_current_hosts_in_room()`, I'm not personally convinced it's contributing to the seriousness of this situation(but included here for completeness).
Images:


UPDATE: Additional information from the other side of the slash in the title

The large spike in traffic caused by queueing and then sending all those requests looks like this:

Start in synapse/handlers/presence.py around FEDERATION_PING_INTERVAL and FEDERATION_TIMEOUT, then trace how the replication notifier queues federation sending requests. Check the get_current_hosts_in_room() database activity during the spike. Done means identifying the cause of the periodic CPU and federation-traffic surge and defining a verified mitigation.

Written by the indexing model from the issue text.