element-hq / element-hq/dendrite
dendrite server suddenly goes completely silent
- Dominant language
- Go
- Stars
- 967
- Forks
- 102
- PR merge metrics
- No merged PRs in 30d
Description
*This issue was originally created by [**@grisu48**](https://github.com/grisu48) at .*
### Background information
- **Dendrite version or git SHA**: 0.8.7
- **Monolith or Polylith?**: Monolith
- **SQLite3 or Postgres?**: Postgres
- **Running in Docker?**: podman
- **`go version`**: (container build-in)
- **Client used (if applicable)**: Element Web and Android
### Description
Since a server reboot this morning I cannot send messages anymore via Element:

The server was running fine for the last weeks without any noticeable issues.
In the dendrite log I find error messages like the following:
```time="2022-06-09T07:34:13.066121909Z" level=error msg="SendEvents failed" error="InputRoomEventsResponse: context canceled" req.id=JaqeWM97Kqaz req.method=PUT req.path="/_matrix/client/r0/rooms/!d8rmAZMAod6n7EsZ:m.feldspaten.org/send/m.room.encrypted/m1654759992909.1" user_id="@phoenix:m.feldspaten.org"
time="2022-06-09T07:34:51.751232393Z" level=error msg="SendEvents failed" error="InputRoomEventsResponse: context canceled" req.id=wV81oVBTTPge req.method=PUT req.path="/_matrix/client/r0/rooms/!d8rmAZMAod6n7EsZ:m.feldspaten.org/send/m.room.encrypted/$local.c2f3d6f8-eb95-42b3-9ff3-38d8e179e2a9" user_id="@phoenix:m.feldspaten.org"
time="2022-06-09T07:35:15.652729793Z" level=error msg="SendEvents failed" error="InputRoomEventsResponse: context canceled" req.id=3Z5ZPJdx0DkE req.method=PUT req.path="/_matrix/client/r0/rooms/!d8rmAZMAod6n7EsZ:m.feldspaten.org/send/m.room.encrypted/m1654759992909.1" user_id="@phoenix:m.feldspaten.org"
time="2022-06-09T07:36:20.461579016Z" level=error msg="SendEvents failed" error="InputRoomEventsResponse: context canceled" req.id=ZRaKTs4mqVqg req.method=PUT req.path="/_matrix/client/r0/rooms/!d8rmAZMAod6n7EsZ:m.feldspaten.org/send/m.room.encrypted/m1654759992909.1" user_id="@phoenix:m.feldspaten.org"
time="2022-06-09T07:36:41.052252332Z" level=info msg="Executing UpdateUserDailyVisits"
time="2022-06-09T07:37:28.839353423Z" level=error msg="SendEvents failed" error="InputRoomEventsResponse: context canceled" req.id=AbPuX72a3rSu req.method=PUT req.path="/_matrix/client/r0/rooms/!d8rmAZMAod6n7EsZ:m.feldspaten.org/send/m.room.encrypted/m1654759992909.1" user_id="@phoenix:m.feldspaten.org"
time="2022-06-09T07:38:45.475754429Z" level=error msg="SendEvents failed" error="InputRoomEventsResponse: context canceled" req.id=u5zAstAfeuKd req.method=PUT req.path="/_matrix/client/r0/rooms/!d8rmAZMAod6n7EsZ:m.feldspaten.org/send/m.room.encrypted/m1654759992909.1" user_id="@phoenix:m.feldspaten.org"
```
### Steps to reproduce
* No concrete reproducer identified
I'm also attaching the server log after the reboot here: [log.txt](https://github.com/matrix-org/dendrite/files/8868030/log.txt)
Contributor guide
Research direction
Start with the attached server log and the repeated SendEvents failures for PUT /_matrix/client/r0/rooms/.../send/m.room.encrypted, tracing the InputRoomEventsResponse: context canceled error. Compare behavior before and after the reboot, using the reported Monolith, Postgres, Podman, and Dendrite 0.8.7 setup; done means identifying a reproducible cause or concrete remediation for messages becoming unsendable.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, postgres
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100