[synapse-only] Tracking missing to_device messages

This issue has been migrated from [#11201](https://github.com/matrix-org/synapse/issues/11201).

---

I've been observing some problems tracking down problems relating to missing to_device messages, resulting in broken E2EE and other problems.

However, i think it's good to at least be able to advertise to administrators when these gaps in messages occur.

This feature will specifically track messages that go missing over federation.

---

The tracking can be done through a header sent with `/send`. For tracking to start, both ends must have acknowledged the feature before tracking starts.

A server will send a header in the following format;

`X-Synapse-ToDevice-Tracking: SESS=1234567890ABCDEF,COUNT=12345`

In cases where the other side hasn't acknowledged a session, `COUNT` is omitted, and to_device messages arent counted.

When a server gets an incoming header with a new `SESS`, it has to send an outgoing `SESS` with `COUNT=0`, no matter how many to_device messages exist in the current transaction. When the other side also has sent `COUNT=0`, counting begins.

At this point, synapse should issue a warning (and possibly update a gauge metric) when:

`incoming_header_count - last_count - amount_of_to_device_messages_in_transaction != 0`

In other words; when the counter saved in the database for the last transaction, plus the amount of to_device messages in the current transaction, doesn't equal the `COUNT` number issued in the header.

---

This can provide a crude way of *at least* tracking how prevalent the problem is, and possibly provide heuristic logging of when high amounts of to_device messages are lost.

This doesn't track the to_device messages lost in internal processing (unless that also affects storing the last `COUNT` in database).

Tacking on the incoming count (for two-way metrics exchange) is tricky, as federation requests can be sent and received asynchronously, so this only provides a "sum check" for incoming to_device messages.

---

This is generally "out of spec" and limited to synapse at the moment, I think this shouldn't be permanent, and only needed to sus out problems pertaining with missing to_device messages.