matrix-org / matrix-org/matrix-rust-sdk
Message Retention Purging Strategy
- Dominant language
- Rust
- Stars
- 2.3k
- Forks
- 500
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 106
Description
This is a focused sub topic of #6713.
For message retention, we need to purge expired message events somehow. My original intention was to migrate the dbs to index the message's timestamp and run a `DELETE WHERE timestamp < expiration` style query, but when discussed with @stefanceriu he raised some concerns. I can't recall all of them, but I believe it revolved around there being multiple different stores used for different purposes.
I'm now proposing that I'd approach this instead by iterating over events from the oldest to the newest, stopping once we encounter a message that is not expired. The added benefit here is that we can inspect each message as we encounter it to also clear up local media caches.
I'd intend to implement by modifying the EventCacheStore to gain `load_first_chunk` and `load_next_chunk` methods, such that we can start from the oldest events and work towards the newest. The biggest catch, however, is that the IndexedDB for web is missing the `previous_id` column in its table to be able to find the first chunk with a simple `WHERE previous_id is NULL`, and instead need to scan the whole table to find it. So this plan would also entail a migration to add the `previous_id` column (and a table wipe to reset).
So, I'm soliciting feedback on this plan, but would be open to other suggestions as well.
Contributor guide
Research direction
Start with #6713 and the EventCacheStore implementations, then inspect the web IndexedDB table and schema, including the missing previous_id column. Before coding, resolve the retention strategy and multiple-store concerns; done should be an agreed purge approach covering migration or table reset implications and local media-cache cleanup.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100