rust-lang / rust-lang/docs.rs

Use hard memory limits for the daemon, not soft limits

Open
#1,368 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

A-admin
Dominant language
Rust
Stars
1.2k
Forks
232
Avg merge
11h 35m
Merged PRs (30d)
60

Description

Right now, if we run out of memory, the server becomes more and more unresponsive until it stops being usable. If we're lucky, the OOM killer eventually notices and kills the process; if not, we have to do a hard-restart of the instance. Instead of using the default soft limit of "when the OOM killer notices", we should have a hard limit - that way the server restarts immediately instead of being down until someone has time to investigate.

systemd has a way to control this with MemoryMax=XXX (https://www.freedesktop.org/software/systemd/man/systemd.resource-control.html). Unfortunately, because docker does shenanigans with cgroups, it doesn't propagate into docker containers. However, since we already limit the memory of docker containers to 3 GB by default, we can just set the MemoryMax 3 GB lower than it would be otherwise. Note that for crates with a raised memory limit, we'll avoid an OOM through https://github.com/rust-lang/docs.rs/pull/1279, which just won't start the build.

We have about 15.5 GB of memory on the prod instance, so I'd suggest a 12 GB hard limit.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the production systemd unit and Docker memory-limit configuration. Compare the existing 3 GB container default with any raised limits, then verify that the daemon receives a 12 GB hard MemoryMax and that the container limits remain compatible.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, rust
Domain
devops, infrastructure
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.