Use hard memory limits for the daemon, not soft limits
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 1.2k
- Forks
- 232
- Avg merge
- 11h 35m
- Merged PRs (30d)
- 60
Description
Right now, if we run out of memory, the server becomes more and more unresponsive until it stops being usable. If we're lucky, the OOM killer eventually notices and kills the process; if not, we have to do a hard-restart of the instance. Instead of using the default soft limit of "when the OOM killer notices", we should have a hard limit - that way the server restarts immediately instead of being down until someone has time to investigate.
systemd has a way to control this with MemoryMax=XXX (https://www.freedesktop.org/software/systemd/man/systemd.resource-control.html). Unfortunately, because docker does shenanigans with cgroups, it doesn't propagate into docker containers. However, since we already limit the memory of docker containers to 3 GB by default, we can just set the MemoryMax 3 GB lower than it would be otherwise. Note that for crates with a raised memory limit, we'll avoid an OOM through https://github.com/rust-lang/docs.rs/pull/1279, which just won't start the build.
We have about 15.5 GB of memory on the prod instance, so I'd suggest a 12 GB hard limit.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the production systemd unit and Docker memory-limit configuration. Compare the existing 3 GB container default with any raised limits, then verify that the daemon receives a 12 GB hard MemoryMax and that the container limits remain compatible.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, rust
- Domain
- devops, infrastructure
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100