element-hq / element-hq/dendrite
Backfilling on `/messages` causes odd behavior
- Dominant language
- Go
- Stars
- 965
- Forks
- 101
- PR merge metrics
- No merged PRs in 30d
Description
*This issue was originally created by [**@puckipedia**](https://github.com/puckipedia) at .*
### Background information
- **Dendrite version or git SHA**: 633ca06eb9f7652a6c4be04b3ffe8950419a8ee3 (0.8.3rc1)
- **Monolith or Polylith?**: Monolith
- **SQLite3 or Postgres?**: Postgres 14
- **Running in Docker?**: No
- **`go version`**: 1.16.7
- **Client used (if applicable)**: Fractal, but replicates with `curl`
### Description
- **What** is the problem: When using a filter on the `/messages` endpoint that doesn't match all messages (e.g. `?from=...&dir=b&limit=40&filter={"types":["nope"]}`) Dendrite skips over a large chunk of events, as it assumes that retrieving any amount of messages less than the limit is incorrect, and then adds the wrongly backfilled events, unfiltered, into the response.
- **Who** is affected: Uncertain.
- **How** is this bug manifesting: Scrolling upwards in rooms works inconsistently (Fractal has large jumps of >1 year in event age)
- **When** did this first appear: Appeared in the first room I joined Dendrite into, a few days ago, on 0.8.2
### Steps to reproduce
- Join Dendrite to a room with (multiple?) unfetchable backwards extremities (in my case, this was `#flatpak:matrix.org`, but it presumably applies to any rooms with non-fully-readable room history)
- Send a request for message scrollback from a range that has been backfilled previously, with a filter that doesn't apply to all messages
- Notice that Dendrite is sending a backfill request that is unneeded when no filter is applied, and has a completely incorrect `end` topological pointer.
- Notice that the response contains the backfilled events, even though they should not show up in the room's history at this point in the topology
Contributor guide
Research direction
Start at the /messages endpoint handling the reproduced curl request with dir=b, limit=40, and a non-matching types filter. Trace how filtered results determine whether backfill is requested and how the end topological pointer is set. Done means no unnecessary backfill occurs, the end pointer is correct, and backfilled events do not bypass the filter.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, postgresql
- Domain
- api, backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100