matrix-org / matrix-org/matrix-rust-sdk

Memory usage of linked chunks can explode over time

Open
#6,485 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
2.3k
Forks
500
Avg merge
1d 16h
Merged PRs (30d)
106

Description

We do have a mechanism to unload a room's linked chunk from memory, and only keep the latest chunk. When a room linked chunk starts to be observed in real-time, for instance by the timeline, (via linked chunk updates, or via the mapped vector updates), we increase a counter of active observers. When one observer is gone, the counter drops by one; when there's no more observers, the room linked chunk is unloaded from memory automatically.

Whenever we get _any_ new event related to a room (even ephemeral ones), the room's linked chunk will be reloaded from storage into memory. Usually that's fine, because only the latest chunk will be reloaded; new events/gaps may add new chunks, but there shouldn't be a lot of chunks active at the same time for a given room. However, small streams make great rivers, and having this happen for many rooms in sync could lead to having a lot of dormant rooms in the background, that are not being observed (so, they won't benefit from the automatic unloading), and that slowly fill up memory.

I am not sure this is a problem in practice at this point. But with the MVP automatic pagination that landed recently, the problem is made much worse: a single new read receipt for a room can lead to the room being fully paginated, in the worst case (e.g. if the read receipt points to one of the first events), and multiplying this by the number of different rooms could lead to lots of memory being used overall.

The motivation for this issue would be to have some *smart* system around that would automatically unload in-memory linked chunks, even those that aren't observed, so as to make sure the memory usage is as minimal as it can be. However, there might be another path, which is to implement automatic pagination that happens only on disk, in which case implementing such an automatic unloading mechanism might be overkill.

Contributor guide

Open the contributing guide

Research direction

Start by tracing linked-chunk loading and unloading, active-observer counting, and automatic pagination triggered by read receipts. Compare observer-independent unloading with disk-only pagination, then define a concrete approach whose completion criterion is bounded memory use across dormant rooms.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.