element-hq / element-hq/synapse

createRoom succeeded, but returned 500, due to replication lag

Open
#8,625 0 comments 0 reactions 0 assignees View on GitHub
A-Create-Room S-Minor T-Defect Z-Read-After-Write
Dominant language
Python
Stars
4.6k
Forks
600
Avg merge
5d 22h
Merged PRs (30d)
51

Description

This issue has been migrated from [#8625](https://github.com/matrix-org/synapse/issues/8625).

---

https://github.com/vector-im/element-web/issues/15512#issuecomment-713575675 reports a `createRoom` request returning a 500.

The server-side logs say:

```
2020-10-21 13:29:51,969 - synapse.replication.tcp.client - 232 - INFO - POST-39220 - Waiting for repl stream 'events' to reach 1605923815
...
2020-10-21 13:30:21,987 - synapse.http.server - 85 - ERROR - POST-39220 - Failed handle request via 'RoomCreateRestServlet':
Capture point (most recent call last):
...
twisted.internet.defer.TimeoutError: Timed out after 30s
2020-10-21 13:30:22,015 - synapse.access.http.8080 - 311 - INFO - POST-39220 - 109.175.168.80 - 8080 - {@webdevguru_test13:matrix.org} Processed request: 39.142sec/-0.000sec (0.121sec, 0.012sec) (0.241sec/0.546sec/58) 55B 500 "POST /_matrix/client/r0/createRoom HTTP/1.1" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86
...
2020-10-21 13:31:30,491 - synapse.replication.tcp.handler - 543 - INFO - process-replication-data-4 - Caught up with stream 'events' to 1605923815
```

It seems that matrix.org was still catching up after an earlier outage at this time:

![image](https://user-images.githubusercontent.com/1389908/96856120-7b662700-1455-11eb-820f-c8712f4a594c.png)

I'm not sure exactly what can be done about it, but it's certainly unsatisfactory that the room creation completes and then we return a 500 anyway.

Contributor guide

Open the contributing guide

Research direction

Start at the RoomCreateRestServlet handling the createRoom endpoint and trace the replication wait shown in the logs. Investigate how the 30-second timeout is handled when the events stream is still catching up. Done should address the reported case so a completed room creation does not result in an erroneous 500 response.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.