element-hq / element-hq/synapse

Process EDUs in the background

Open
#5,175 0 comments 0 reactions 0 assignees View on GitHub
A-to-device-messages T-Enhancement z-p3
Dominant language
Python
Stars
4.6k
Forks
600
Avg merge
5d 22h
Merged PRs (30d)
51

Description

This issue has been migrated from [#5175](https://github.com/matrix-org/synapse/issues/5175).

---

Author's note: Sorry about the short story here. The background information feels important to describe one possible use case for implementing this feature, even if it is rare.

The spec doesn't actually say that they should be handled in sync with the request ( https://matrix.org/docs/spec/server_server/r0.1.1.html#put-matrix-federation-v1-send-txnid ), which may be a bug itself, however it would be nice to have EDUs processed async to PDUs.

For a bit of background: t2bot.io has ignored each and every EDU for the last year for performance reasons. Nothing on the server has used incoming EDUs for anything, so it was just a wasted amount of processing power to deal with them.

Due to an unrelated problem, the server fell behind on federation and was catching back up. During that recovery period, I decided to re-enable EDU processing for encryption-related stuff for a future offering on the server (namely bots which are e2e-capable).

Device list updates then started pouring in, taking on average 300s each to process. This resulted in PDUs being even further delayed because replication would often time out between the federation reader and main process for these device list updates. When the timeout occurred, the federation reader would 500 the transaction request and the server would re-send it.

Although it's likely my fault for device list updates taking 300s to process (given they weren't touched for a year), it would be nice if EDUs could be handled async to PDUs so that PDUs can continue to go through the server without being blocked by the less-important EDUs (in my case they are less important, at least).

Also worth noting that CPU usage was fantastically small when running into 300s device list updates processing times. Normally I'd expect some amount of torture on the system for request times like that, but instead it looked like traffic was back to small homeserver levels:
![image](https://user-images.githubusercontent.com/1190097/57576642-04380100-7422-11e9-8bb2-8c9e220589dc.png)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.