element-hq / element-hq/synapse
State resolution of a large room but small deltas caused issues on event persister
- Dominant language
- Python
- Stars
- 4.6k
- Forks
- 600
- Avg merge
- 5d 22h
- Merged PRs (30d)
- 51
Description
This issue has been migrated from [#12378](https://github.com/matrix-org/synapse/issues/12378).
---
This led to all events being persisted on that worker to take a while to persist, heavily impacting our apdex. The room in question had 30K events, but the forks were very recent. The `3 biggest rooms for state-res` metrics had numbers such as `13.461s` CPU time and `6.54011s` DB time. During this period the reactor tick time ended up being large, which is likely what caused the delays persisting events in other rooms.
Some things we can do here:
* implement a delta state resolution algorithm
* speed up state res
* try and have state res yield to the reactor more often, so that it doesn't block the reactor
Contributor guide
Assessment
This issue has not been assessed yet.