element-hq / element-hq/dendrite

Backfilling on `/messages` causes odd behavior

Open
#2,433 6 comments 0 reactions 0 assignees View on GitHub
C-Sync-API F-Filters possibly-fixed T-Defect
Dominant language
Go
Stars
965
Forks
101
PR merge metrics
No merged PRs in 30d

Description

*This issue was originally created by [**@puckipedia**](https://github.com/puckipedia) at .*

### Background information

- **Dendrite version or git SHA**: 633ca06eb9f7652a6c4be04b3ffe8950419a8ee3 (0.8.3rc1)
- **Monolith or Polylith?**: Monolith
- **SQLite3 or Postgres?**: Postgres 14
- **Running in Docker?**: No
- **`go version`**: 1.16.7
- **Client used (if applicable)**: Fractal, but replicates with `curl`

### Description

- **What** is the problem: When using a filter on the `/messages` endpoint that doesn't match all messages (e.g. `?from=...&dir=b&limit=40&filter={"types":["nope"]}`) Dendrite skips over a large chunk of events, as it assumes that retrieving any amount of messages less than the limit is incorrect, and then adds the wrongly backfilled events, unfiltered, into the response.
- **Who** is affected: Uncertain.
- **How** is this bug manifesting: Scrolling upwards in rooms works inconsistently (Fractal has large jumps of >1 year in event age)
- **When** did this first appear: Appeared in the first room I joined Dendrite into, a few days ago, on 0.8.2

### Steps to reproduce
- Join Dendrite to a room with (multiple?) unfetchable backwards extremities (in my case, this was `#flatpak:matrix.org`, but it presumably applies to any rooms with non-fully-readable room history)
- Send a request for message scrollback from a range that has been backfilled previously, with a filter that doesn't apply to all messages
- Notice that Dendrite is sending a backfill request that is unneeded when no filter is applied, and has a completely incorrect `end` topological pointer.
- Notice that the response contains the backfilled events, even though they should not show up in the room's history at this point in the topology

Contributor guide

Open the contributing guide

Research direction

Start at the /messages endpoint handling the reproduced curl request with dir=b, limit=40, and a non-matching types filter. Trace how filtered results determine whether backfill is requested and how the end topological pointer is set. Done means no unnecessary backfill occurs, the end pointer is correct, and backfilled events do not bypass the filter.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, postgresql
Domain
api, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.