matrix-org / matrix-org/matrix-rust-sdk
Investigate slow performance of event cache
- Dominant language
- Rust
- Stars
- 2.3k
- Forks
- 500
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 106
Description
- The event cache store uses a cross-process lock to make sure that there's at most one process writing to its DB.
- At another layer, the sqlite backend uses a [write lock](https://github.com/matrix-org/matrix-rust-sdk/pull/5382) to prevent against bad `database busy` errors, which has proven super effective.
Unfortunately, it seems that the interaction of both can dramatically slow down the SDK: taking the cross-process lock will read/write into the DB into a "leases" database, making use of the sqlite write lock.
This is even worse, because the event cache store is also used for medias at the moment; ideally it'd be split off, and there's https://github.com/matrix-org/matrix-rust-sdk/issues/5410 for that.
In the meanwhile: we should also explore ways to reduce contention around the cross-process lock, so as to keep it as long as possible, maybe for a full sync update, and pass it down to methods that need it, instead of taking it randomly as we see fit. We might need to be careful around the dining philosopher's problem, though; if our locks are more fine-grained, then there are more chances we're taking multiple ones at the same time.
## See also
- https://github.com/matrix-org/matrix-rust-sdk/pull/5426 tried to process updates in parallel in the event cache, which led to a slowdown; we never had time to investigate why.
- https://github.com/matrix-org/matrix-rust-sdk/issues/5410 would help to separate responsibilities in the event cache, and likely lower contention around the cross-process / sqlite write locks.
- https://github.com/matrix-org/matrix-rust-sdk/issues/5500 would block the sync updates on the event cache updates, which might make the problem more prominent (i.e. easier to identify, but also may slow down everyone).
Contributor guide
Research direction
No source files or tests are named. Start by profiling the event cache store and the interaction between its cross-process lease lock and SQLite write lock, then compare the parallel-update behavior described in pull request 5426 and the related issues. Done should identify the contention and produce an agreed, scoped approach for reducing it without introducing lock-ordering problems.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, sqlite
- Domain
- backend, database, performance
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100