element-hq / element-hq/dendrite
syncserver: double-check backfills for duplicates
- Dominant language
- Go
- Stars
- 965
- Forks
- 101
- PR merge metrics
- No merged PRs in 30d
Description
*This issue was originally created by [**@kegsay**](https://github.com/kegsay) at .*
When a client hits `/messages`, we try to get earlier messages from the DB. If we can't get enough, we backfill from other servers. Whenever we hit other servers, we need to make sure we protect ourselves from their bugs/problems.
If a remote server sends a duplicate event (same event ID/depth) for whatever reason, we insert the event into the `syncapi_output_room_events_topology` table, which correctly de-dupes based on `ON CONFLICT (topological_position, room_id) DO UPDATE SET event_id = $1`, but the request that initiated the backfill receives the duplicate event.
When we know we need to backfill, we should do so and then **re-run** the `/messages` request to pull from the DB, rather than return whatever events the remote server gave us, as the DB has extra checks in place like the aforementioned de-dupe logic.
This pattern of "hit remote servers to get data then re-run our internal logic which we trust" is probably something we should do in more places too!
Contributor guide
Assessment
This issue has not been assessed yet.