element-hq / element-hq/dendrite

syncserver: double-check backfills for duplicates

Open
#932 0 comments 0 reactions 0 assignees View on GitHub
C-Sync-API T-Defect
Dominant language
Go
Stars
965
Forks
101
PR merge metrics
No merged PRs in 30d

Description

*This issue was originally created by [**@kegsay**](https://github.com/kegsay) at .*

When a client hits `/messages`, we try to get earlier messages from the DB. If we can't get enough, we backfill from other servers. Whenever we hit other servers, we need to make sure we protect ourselves from their bugs/problems.

If a remote server sends a duplicate event (same event ID/depth) for whatever reason, we insert the event into the `syncapi_output_room_events_topology` table, which correctly de-dupes based on `ON CONFLICT (topological_position, room_id) DO UPDATE SET event_id = $1`, but the request that initiated the backfill receives the duplicate event.

When we know we need to backfill, we should do so and then **re-run** the `/messages` request to pull from the DB, rather than return whatever events the remote server gave us, as the DB has extra checks in place like the aforementioned de-dupe logic.

This pattern of "hit remote servers to get data then re-run our internal logic which we trust" is probably something we should do in more places too!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.