matrix-org / matrix-org/matrix-rust-sdk

event cache: be more conservative when adding duplicated events to a linked chunk

Open
#4,563 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
2.3k
Forks
500
Avg merge
1d 16h
Merged PRs (30d)
106

Description

When we're adding new events into the linked chunk, we try to minimize storing of back-pagination prev-batch tokens:

  1. when pushing to the tail, if all events have been duplicated, independently of ordering, we won't deduplicate/remove them, and we'll not store the prev-batch token.
  2. when inserting a chunk of events from a recent back-pagination, and they've all been deduplicated, we won't deduplicate/remove them, and we'll not store the prev-batch token.

Unfortunately, this is not as simple as that: we should also check the orderings of the events that have been deduplicated, in particular that they're indeed ordered the same way they're ordered in the linked chunk representation. If that's not the case, then we should keep the prev-batch token and keep on iterating until we get the same events in the same ordering.

Related to #3280.

cc @Hywan

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the event-cache logic that adds events to linked chunks, including tail pushes and recent back-pagination insertion. Compare cases where duplicated events have different orderings, and verify completion when the implementation preserves the prev-batch token until matching event order is reached.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.