element-hq / element-hq/dendrite

dendrite server suddenly goes completely silent

Open
#2,524 14 comments 0 reactions 0 assignees View on GitHub
C-NATS O-Uncommon S-Critical T-Defect
Dominant language
Go
Stars
967
Forks
102
PR merge metrics
No merged PRs in 30d

Description

*This issue was originally created by [**@grisu48**](https://github.com/grisu48) at .*

### Background information
- **Dendrite version or git SHA**: 0.8.7
- **Monolith or Polylith?**: Monolith
- **SQLite3 or Postgres?**: Postgres
- **Running in Docker?**: podman
- **`go version`**: (container build-in)
- **Client used (if applicable)**: Element Web and Android

### Description

Since a server reboot this morning I cannot send messages anymore via Element:

![image](https://user-images.githubusercontent.com/1527829/172792057-3dcbaa6b-ba0f-43a5-9a74-052278e9ad63.png)

The server was running fine for the last weeks without any noticeable issues.

In the dendrite log I find error messages like the following:

```time="2022-06-09T07:34:13.066121909Z" level=error msg="SendEvents failed" error="InputRoomEventsResponse: context canceled" req.id=JaqeWM97Kqaz req.method=PUT req.path="/_matrix/client/r0/rooms/!d8rmAZMAod6n7EsZ:m.feldspaten.org/send/m.room.encrypted/m1654759992909.1" user_id="@phoenix:m.feldspaten.org"
time="2022-06-09T07:34:51.751232393Z" level=error msg="SendEvents failed" error="InputRoomEventsResponse: context canceled" req.id=wV81oVBTTPge req.method=PUT req.path="/_matrix/client/r0/rooms/!d8rmAZMAod6n7EsZ:m.feldspaten.org/send/m.room.encrypted/$local.c2f3d6f8-eb95-42b3-9ff3-38d8e179e2a9" user_id="@phoenix:m.feldspaten.org"
time="2022-06-09T07:35:15.652729793Z" level=error msg="SendEvents failed" error="InputRoomEventsResponse: context canceled" req.id=3Z5ZPJdx0DkE req.method=PUT req.path="/_matrix/client/r0/rooms/!d8rmAZMAod6n7EsZ:m.feldspaten.org/send/m.room.encrypted/m1654759992909.1" user_id="@phoenix:m.feldspaten.org"
time="2022-06-09T07:36:20.461579016Z" level=error msg="SendEvents failed" error="InputRoomEventsResponse: context canceled" req.id=ZRaKTs4mqVqg req.method=PUT req.path="/_matrix/client/r0/rooms/!d8rmAZMAod6n7EsZ:m.feldspaten.org/send/m.room.encrypted/m1654759992909.1" user_id="@phoenix:m.feldspaten.org"
time="2022-06-09T07:36:41.052252332Z" level=info msg="Executing UpdateUserDailyVisits"
time="2022-06-09T07:37:28.839353423Z" level=error msg="SendEvents failed" error="InputRoomEventsResponse: context canceled" req.id=AbPuX72a3rSu req.method=PUT req.path="/_matrix/client/r0/rooms/!d8rmAZMAod6n7EsZ:m.feldspaten.org/send/m.room.encrypted/m1654759992909.1" user_id="@phoenix:m.feldspaten.org"
time="2022-06-09T07:38:45.475754429Z" level=error msg="SendEvents failed" error="InputRoomEventsResponse: context canceled" req.id=u5zAstAfeuKd req.method=PUT req.path="/_matrix/client/r0/rooms/!d8rmAZMAod6n7EsZ:m.feldspaten.org/send/m.room.encrypted/m1654759992909.1" user_id="@phoenix:m.feldspaten.org"
```

### Steps to reproduce

* No concrete reproducer identified

I'm also attaching the server log after the reboot here: [log.txt](https://github.com/matrix-org/dendrite/files/8868030/log.txt)

Contributor guide

Open the contributing guide

Research direction

Start with the attached server log and the repeated SendEvents failures for PUT /_matrix/client/r0/rooms/.../send/m.room.encrypted, tracing the InputRoomEventsResponse: context canceled error. Compare behavior before and after the reboot, using the reported Monolith, Postgres, Podman, and Dendrite 0.8.7 setup; done means identifying a reproducible cause or concrete remediation for messages becoming unsendable.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, postgres
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.