element-hq / element-hq/synapse

Backfill should gracefully handle the case where no one has the complete state for a given event

Open
#10,764 0 comments 0 reactions 0 assignees View on GitHub
A-Federation A-Messages-Endpoint T-Enhancement
Dominant language
Python
Stars
4.6k
Forks
600
Avg merge
5d 22h
Merged PRs (30d)
51

Description

This issue has been migrated from [#10764](https://github.com/matrix-org/synapse/issues/10764).

---

*This issue is spawning from https://github.com/matrix-org/synapse/pull/10566#discussion_r700713381 and summarizes the problem/conversation there. Thanks @erikjohnston and @richvdh for the insights on this!*

---

We initially used a `fake_prev_event_id` for MSC2716 to make a chain of events float outside of the normal DAG but have since [opted to use empty `prev_events=[]`](https://github.com/matrix-org/synapse/pull/11114). This issue still valid for the general case where no servers have the state for a given event though.

The refactor in https://github.com/matrix-org/synapse/pull/10645 changed the way we handle fetching state for backfills. Before we'd just get the state at the last event, whereas now we try and get the state for each `prev_event` (including the fake event), which obviously fails. This means that federated homeservers no longer accept the insertion event because it can't fetch the `fake_prev_event_id` state (`Error attempting to resolve state at missing prev_events`, `ERROR 403: We can't get valid state history.`).

The goal of the code that's rejecting the fake event is protecting against an attack where a remote server could arbitrarily replace the servers view of the current state of the room. **But the mitigation we use is a sledgehammer and we should replace it with something more intelligent that both mitigates the attack and gracefully handles missing events.** (related to https://github.com/matrix-org/synapse/issues/8451)

## Potential solutions

To be determined (TBD).

## Reproduction case

1. Create an event which references a made up `prev_event_id`
1. Try to backfill it from federated homeserver

```sh
2021-09-01 20:19:21,560 - synapse.handlers.federation_event - 746 - WARNING - GET-5-$hbHXgulNKlmg9mkGQiBH5SSgq2NL4GO70sIIfHCa-YY - Error attempting to resolve state at missing prev_events
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation_event.py", line 711, in _resolve_state_at_missing_prevs
remote_state = await self._get_state_after_missing_prev_event(
File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation_event.py", line 777, in _get_state_after_missing_prev_event
) = await self.federation_client.get_room_state_ids(
File "/usr/local/lib/python3.8/site-packages/synapse/federation/federation_client.py", line 370, in get_room_state_ids
result = await self.transport_layer.get_room_state_ids(
File "/usr/local/lib/python3.8/site-packages/synapse/federation/transport/client.py", line 71, in get_room_state_ids
return await self.client.get_json(
File "/usr/local/lib/python3.8/site-packages/synapse/http/matrixfederationclient.py", line 1001, in get_json
response = await self._send_request_with_optional_trailing_slash(
File "/usr/local/lib/python3.8/site-packages/synapse/http/matrixfederationclient.py", line 385, in _send_request_with_optional_trailing_slash
response = await self._send_request(request, **send_request_args)
File "/usr/local/lib/python3.8/site-packages/synapse/http/matrixfederationclient.py", line 631, in _send_request
raise exc
synapse.api.errors.HttpResponseException: 404: Not Found
```

---

- https://github.com/matrix-org/internal-config/issues/228

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.