element-hq / element-hq/synapse
Backfill should gracefully handle the case where no one has the complete state for a given event
- Dominant language
- Python
- Stars
- 4.6k
- Forks
- 600
- Avg merge
- 5d 22h
- Merged PRs (30d)
- 51
Description
This issue has been migrated from [#10764](https://github.com/matrix-org/synapse/issues/10764).
---
*This issue is spawning from https://github.com/matrix-org/synapse/pull/10566#discussion_r700713381 and summarizes the problem/conversation there. Thanks @erikjohnston and @richvdh for the insights on this!*
---
We initially used a `fake_prev_event_id` for MSC2716 to make a chain of events float outside of the normal DAG but have since [opted to use empty `prev_events=[]`](https://github.com/matrix-org/synapse/pull/11114). This issue still valid for the general case where no servers have the state for a given event though.
The refactor in https://github.com/matrix-org/synapse/pull/10645 changed the way we handle fetching state for backfills. Before we'd just get the state at the last event, whereas now we try and get the state for each `prev_event` (including the fake event), which obviously fails. This means that federated homeservers no longer accept the insertion event because it can't fetch the `fake_prev_event_id` state (`Error attempting to resolve state at missing prev_events`, `ERROR 403: We can't get valid state history.`).
The goal of the code that's rejecting the fake event is protecting against an attack where a remote server could arbitrarily replace the servers view of the current state of the room. **But the mitigation we use is a sledgehammer and we should replace it with something more intelligent that both mitigates the attack and gracefully handles missing events.** (related to https://github.com/matrix-org/synapse/issues/8451)
## Potential solutions
To be determined (TBD).
## Reproduction case
1. Create an event which references a made up `prev_event_id`
1. Try to backfill it from federated homeserver
```sh
2021-09-01 20:19:21,560 - synapse.handlers.federation_event - 746 - WARNING - GET-5-$hbHXgulNKlmg9mkGQiBH5SSgq2NL4GO70sIIfHCa-YY - Error attempting to resolve state at missing prev_events
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation_event.py", line 711, in _resolve_state_at_missing_prevs
remote_state = await self._get_state_after_missing_prev_event(
File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation_event.py", line 777, in _get_state_after_missing_prev_event
) = await self.federation_client.get_room_state_ids(
File "/usr/local/lib/python3.8/site-packages/synapse/federation/federation_client.py", line 370, in get_room_state_ids
result = await self.transport_layer.get_room_state_ids(
File "/usr/local/lib/python3.8/site-packages/synapse/federation/transport/client.py", line 71, in get_room_state_ids
return await self.client.get_json(
File "/usr/local/lib/python3.8/site-packages/synapse/http/matrixfederationclient.py", line 1001, in get_json
response = await self._send_request_with_optional_trailing_slash(
File "/usr/local/lib/python3.8/site-packages/synapse/http/matrixfederationclient.py", line 385, in _send_request_with_optional_trailing_slash
response = await self._send_request(request, **send_request_args)
File "/usr/local/lib/python3.8/site-packages/synapse/http/matrixfederationclient.py", line 631, in _send_request
raise exc
synapse.api.errors.HttpResponseException: 404: Not Found
```
---
- https://github.com/matrix-org/internal-config/issues/228
Contributor guide
Assessment
This issue has not been assessed yet.