element-hq / element-hq/synapse

We sometimes backfill on `/messages` requests when we shouldn't.

Open
#10,742 0 comments 0 reactions 0 assignees View on GitHub
A-Federation A-Messages-Endpoint S-Minor T-Defect
Dominant language
Python
Stars
4.6k
Forks
607
Avg merge
5d 22h
Merged PRs (30d)
51

Description

This issue has been migrated from [#10742](https://github.com/matrix-org/synapse/issues/10742).

---

This happens when we use a `from` token that just has a stream ordering part (which clients do the first time they start back paginating in a room after a `/sync` request), as the following function to convert a stream token to a topological token can lead to incorrect results:

https://github.com/matrix-org/synapse/blob/e2481dbe9325321d460037a2efe9b9ea2ac78057/synapse/storage/databases/main/stream.py#L747-L762

This is a particularly big problem for rooms like Matrix HQ, where repeated calls to `/messages` will always trigger a backfill and then time out, even though the server has enough events to return to the client.

The problem manifests when the server receives an old event, so that it ends up with an event with a recent stream ordering but an old depth, causing the above function to return the old depth.

Contributor guide

Open the contributing guide

Research direction

Start at synapse/storage/databases/main/stream.py lines 747-762, focusing on conversion of a stream-only token to a topological token. Reproduce a /messages request after /sync with an old event having recent stream ordering and old depth; done means repeated requests return available events without triggering an unnecessary backfill or timing out.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend-api-design, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.