getsentry / getsentry/self-hosted

snuba-api worker memory recycling is disabled by default, and compose still sets dead UWSGI_* vars after the Granian migration

Open
#4,453 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Shell
Stars
9.6k
Forks
2k
Avg merge
1d 5h
Merged PRs (30d)
15

Description

### Self-Hosted Version

26.6.0

### CPU Architecture

x86_64

### Docker Version

Docker version 29.6.2, build dfc4efb

### Docker Compose Version

Docker Compose version v5.3.1

### Machine Specification

4 vCPU, 32 GB RAM, 100 GB SSD, no swap. Cloud VM, Debian 12 (kernel 6.1.0-38-amd64).

RAM is at the level the docs recommend ("We recommend using 32 GB RAM"). No swap is configured. I want to flag this up front because the issue below describes *unbounded* growth — swap would delay the outcome but cannot bound it, so I don't think swap is the missing piece here.

- [X] My system meets the minimum system requirements of Sentry

### Installation Type

Upgrade

### Steps to Reproduce

1. Run self-hosted 26.6.0 on a low-to-moderate traffic instance.
2. Leave `snuba-api` running for a week or more without restarting it.
3. Watch the RSS of the Granian worker process:

```bash
ps -eo pid,rss,etimes,comm | grep "snuba-api worke"
```

### Expected Result

The `snuba-api` worker recycles at some bounded memory ceiling, the way `web` does — `sentry/sentry.conf.py` sets `reload-on-rss: 600` and `max-worker-lifetime: 86400` for its uWSGI workers, and that service stays flat at ~480 MB per worker indefinitely.

### Actual Result

The `snuba-api` worker grows without bound and never recycles.

**Observed on my instance:** a single worker reached **7.3 GB RSS after 15 days** — roughly 470 MB/day — against a fresh-start baseline of ~197 MB. `VmHWM` equalled `VmRSS`, so the growth was monotonic and never released:

```
Name: snuba-api worke
VmPeak: 12503436 kB
VmHWM: 7305956 kB
VmRSS: 7303732 kB
RssAnon: 7299412 kB
```

That is 22.5% of host RAM in one process. Since no service in `docker-compose.yml` sets `mem_limit`, the kernel eventually fires a **global** OOM kill and picks the largest process on the host rather than the container responsible — I have 6 such events in `journalctl -k`, all `constraint=CONSTRAINT_NONE ... global_oom`, with victims that were bystanders rather than the growing process.

### Root cause in configuration

`snuba-api` has run on **Granian, not uWSGI**, since getsentry/snuba#7566 (`prodeng-605: migrate uwsgi to granian`, Jan 2026). Confirmed on the running container:

```bash
$ grep -c granian /proc//maps
4
$ grep -oE '[^/]*granian[^ ]*' /proc//maps | sort -u
granian/_granian.cpython-313-x86_64-linux-gnu.so
$ grep -ci uwsgi /proc//maps
0
```

Two consequences:

**1. The `UWSGI_*` vars in `docker-compose.yml` are inert.** `x-snuba-defaults` still sets them at lines 116-117 on `master`:

```yaml
UWSGI_MAX_REQUESTS: "10000"
UWSGI_DISABLE_LOGGING: "true"
```

Nothing reads these anymore. `UWSGI_MAX_REQUESTS` in particular looks like it is providing worker recycling, but it has been a no-op for about seven months.

**2. Granian's recycling options are never set, so they default to disabled.** `snuba/settings/settings_self_hosted.py` reads them from the environment:

```python
API_WORKERS = int(env("SNUBA_API_WORKERS", 1))
API_THREADS = int(env("SNUBA_API_THREADS", 8))
API_WORKERS_LIFETIME = int(env("SNUBA_API_WORKERS_LIFETIME")) if ... else None
API_WORKERS_MAX_RSS = int(env("SNUBA_API_WORKERS_MAX_RSS")) if ... else None
```

and `snuba/utils/server.py` passes them to Granian as `workers_lifetime` / `workers_max_rss`. Both are `None` unless the environment supplies them, and `self-hosted`'s compose supplies neither — verified with `docker inspect`, no `SNUBA_API_*` var is present on the container.

These knobs were added by getsentry/snuba#7735, titled *"ref(api): add server settings, allow self-hosted overrides through env"* (Feb 2026) — so they appear to have been built for this deployment, but `self-hosted` never wired them up.

### Suggested fix

In `x-snuba-defaults`, drop the two dead `UWSGI_*` entries and add overridable defaults:

```yaml
SNUBA_API_WORKERS_MAX_RSS: "${SNUBA_API_WORKERS_MAX_RSS:-1024}" # MiB
SNUBA_API_WORKERS_LIFETIME: "${SNUBA_API_WORKERS_LIFETIME:-86400}" # seconds
```

I applied exactly this locally and the worker now starts at ~197 MB with the cap in place. Granian samples RSS every 30s by default and respawns independently of request volume, which matters here: on a low-traffic instance a request-count-based threshold like the old `max-requests: 10000` would rarely be reached anyway.

Worth noting for whoever picks this up: Granian's README warns that *multiple* workers combined with respawn options need kernel >= 5.14 **and** `net.ipv4.tcp_migrate_req` enabled, otherwise queued backlog connections can be dropped during respawn. `SNUBA_API_WORKERS` defaults to 1, so the default path is unaffected — but it argues against raising the worker count as part of any fix here.

I'm not claiming to have located the leak inside Snuba itself; I only have the growth curve. But regardless of where the allocation comes from, self-hosted currently ships with no ceiling on it, and a `mem_limit` on the service would additionally keep an OOM contained to the responsible container instead of taking down a random bystander.

### Event ID

_No response_

Contributor guide

Open the contributing guide

Research direction

Inspect x-snuba-defaults in docker-compose.yml, then read snuba/settings/settings_self_hosted.py and snuba/utils/server.py to trace the worker lifetime and RSS settings into Granian. Use docker inspect or the compose configuration to verify the effective environment. Done means the dead UWSGI_* variables are removed, bounded overridable SNUBA_API_* defaults are present, and the snuba-api worker receives them.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, docker-compose, python
Domain
devops, infrastructure
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
70/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.