element-hq / element-hq/dendrite

NATS JetStream startup message indicates more memory available than soft memory limit set by GOMEMLIMIT

Open
#3,436 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
965
Forks
101
PR merge metrics
No merged PRs in 30d

Description

*This issue was originally created by [**@paigeadelethompson**](https://github.com/paigeadelethompson) at .*

> ​ The Go authors explicitly label GOMEMLIMIT a "soft" limit. That means that the Go runtime does not guarantee that the memory usage will exceed the limit. Instead, it uses it as a target.

Just to be clear I'm not missing that point at all nor should anybody else misunderstand. It actually does fairly good job of minimizing the amount of memory it will try to use but there are some loose ends in Dendrite that will eventually run away and the system (having 2GB total) will ultimately become unusable. It's also not unimaginable that a glimpse into a profiler (maybe pprof) might reveal a couple of things that could be made to work more efficiently.

I really want to say yes, because if nothing else you can limit the work load and refuse work, preempt less important work and as near as I can tell to some extent it already does this but I think the thing that is missing is either a backlog or just outright refusing work that the process doesn't have the capacity for--given a soft memory limit.

Essentially what I have to do because of this is set a hard limit, sort of like an "upper maximum" that will ensure that the process is OOM killed when it finally runs away and makes the whole system unusable. I still believe that Dendrite itself should be able to make the best use of what it has available to it and refuse work gracefully, in general this isn't a hobby that a lot of people are willing to pay money for if even a little so you can reasonably expect most of the people who are going to be using this to want to run on the absolute minimum set of requirements. I gather there are some people who are working on this project who in fact tell you that they're running on just 2GB of memory without a SOFTLIMIT but I would guess the problem with that is they're not actually trying to cruise around and be in more than 1-2 channels.

On an unrelated note and one that is probably out of scope for this reported issue I think a big problem with Matrix is that a lot of people don't know what the hell it's doing and the conclusion that people arrive at is that it's more broken than it actually is; if more instrumentation could be made available through common frontends like element about events (failed/in-flight/succeeded), kind, event duration by room, virtually anything that is measurable that helps people arrive at a conclusion about whether the problem is with them or somebody else would help to eliminate myths about whether or not it works correctly.

Contributor guide

Open the contributing guide

Research direction

The issue names no Dendrite files, tests, or entry points. Start by locating the NATS JetStream startup message and the GOMEMLIMIT handling; clarify whether done means correcting the reported memory value or designing graceful capacity behavior, then verify the agreed behavior with focused coverage.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, performance
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.