element-hq / element-hq/dendrite

"enable_outbound" goes out of control and pegs the CPU

Open
#2,919 10 comments 0 reactions 0 assignees View on GitHub
C-Federation F-Presence T-Defect X-Performance
Dominant language
Go
Stars
965
Forks
101
PR merge metrics
No merged PRs in 30d

Description

*This issue was originally created by [**@IngwiePhoenix**](https://github.com/IngwiePhoenix) at .*

### Background information

- **Dendrite version or git SHA**: 0.10.8+76db8e90
- **Monolith or Polylith?**: Monolith
- **SQLite3 or Postgres?**: Postgres
- **Running in Docker?**: No
- **`go version`**: 1.19.3
- **Client used (if applicable)**: Element, Cinny

### Description

- **What** is the problem: When `presence.enable_outbound` is enabled, my VPS goes under complete load, showing an "Average load" of 40. In `htop`, this means all cores are nothing but red bars. That is, if my SSH connection even survives...
- **Who** is affected: My server (and my VPS' hypervisor, probably)
- **How** is this bug manifesting: `presence.enable_outbound: true` shows a lot of `m.presence` events being sent. My guess: There is no rate limiting in place, meaning it will just infinitly barf them out - and, depending on how many servers/rooms are involved, this can easily become a lot. I am not in `#matrix:matrix.org`, but only in a few small channels - the Dendrite channels amongst them, IPFS, a few old Gitter ones and Telegram 1:1s.
- **When** did this first appear: Today. I was moving between verrsions and upgraded from 0.10.6 to .7 and then .8. Once the bug manifested, I changed to building from `main` in hopes of this being fixed...Well, it's not.

### Steps to reproduce
```
git clone ...
git checkout tags/v0.10.8
mv .git .gitx # Works around a broken build
./build.sh
mv .gitx .git
nano .../dendrite.yaml # set presence.enable_outbound: true
systemctl start matrix-dendrite # I use a SystemD unit
```

Ultimatively, I'd like to be able to see some performance stats logged to tell what exactly is making my server go absolutely crazy. For now, I have this disabled and it seems fine. But for better or for worse, I would like to enable it eventually.

Thanks and kind regards,
Ingwie

Contributor guide

Open the contributing guide

Research direction

Reproduce the issue using Dendrite 0.10.8 or the reported main build with `presence.enable_outbound: true`, then observe CPU usage and outbound `m.presence` events. Start by tracing the presence outbound path and use performance statistics or logs to identify the loop or send rate involved. Done means outbound presence no longer pegs the CPU and its behavior is covered by verification.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.