getsentry / getsentry/self-hosted
snuba-api worker memory recycling is disabled by default, and compose still sets dead UWSGI_* vars after the Granian migration
- Dominant language
- Shell
- Stars
- 9.6k
- Forks
- 2k
- Avg merge
- 1d 5h
- Merged PRs (30d)
- 15
Description
### Self-Hosted Version
26.6.0
### CPU Architecture
x86_64
### Docker Version
Docker version 29.6.2, build dfc4efb
### Docker Compose Version
Docker Compose version v5.3.1
### Machine Specification
4 vCPU, 32 GB RAM, 100 GB SSD, no swap. Cloud VM, Debian 12 (kernel 6.1.0-38-amd64).
RAM is at the level the docs recommend ("We recommend using 32 GB RAM"). No swap is configured. I want to flag this up front because the issue below describes *unbounded* growth — swap would delay the outcome but cannot bound it, so I don't think swap is the missing piece here.
- [X] My system meets the minimum system requirements of Sentry
### Installation Type
Upgrade
### Steps to Reproduce
1. Run self-hosted 26.6.0 on a low-to-moderate traffic instance.
2. Leave `snuba-api` running for a week or more without restarting it.
3. Watch the RSS of the Granian worker process:
```bash
ps -eo pid,rss,etimes,comm | grep "snuba-api worke"
```
### Expected Result
The `snuba-api` worker recycles at some bounded memory ceiling, the way `web` does — `sentry/sentry.conf.py` sets `reload-on-rss: 600` and `max-worker-lifetime: 86400` for its uWSGI workers, and that service stays flat at ~480 MB per worker indefinitely.
### Actual Result
The `snuba-api` worker grows without bound and never recycles.
**Observed on my instance:** a single worker reached **7.3 GB RSS after 15 days** — roughly 470 MB/day — against a fresh-start baseline of ~197 MB. `VmHWM` equalled `VmRSS`, so the growth was monotonic and never released:
```
Name: snuba-api worke
VmPeak: 12503436 kB
VmHWM: 7305956 kB
VmRSS: 7303732 kB
RssAnon: 7299412 kB
```
That is 22.5% of host RAM in one process. Since no service in `docker-compose.yml` sets `mem_limit`, the kernel eventually fires a **global** OOM kill and picks the largest process on the host rather than the container responsible — I have 6 such events in `journalctl -k`, all `constraint=CONSTRAINT_NONE ... global_oom`, with victims that were bystanders rather than the growing process.
### Root cause in configuration
`snuba-api` has run on **Granian, not uWSGI**, since getsentry/snuba#7566 (`prodeng-605: migrate uwsgi to granian`, Jan 2026). Confirmed on the running container:
```bash
$ grep -c granian /proc//maps
4
$ grep -oE '[^/]*granian[^ ]*' /proc//maps | sort -u
granian/_granian.cpython-313-x86_64-linux-gnu.so
$ grep -ci uwsgi /proc//maps
0
```
Two consequences:
**1. The `UWSGI_*` vars in `docker-compose.yml` are inert.** `x-snuba-defaults` still sets them at lines 116-117 on `master`:
```yaml
UWSGI_MAX_REQUESTS: "10000"
UWSGI_DISABLE_LOGGING: "true"
```
Nothing reads these anymore. `UWSGI_MAX_REQUESTS` in particular looks like it is providing worker recycling, but it has been a no-op for about seven months.
**2. Granian's recycling options are never set, so they default to disabled.** `snuba/settings/settings_self_hosted.py` reads them from the environment:
```python
API_WORKERS = int(env("SNUBA_API_WORKERS", 1))
API_THREADS = int(env("SNUBA_API_THREADS", 8))
API_WORKERS_LIFETIME = int(env("SNUBA_API_WORKERS_LIFETIME")) if ... else None
API_WORKERS_MAX_RSS = int(env("SNUBA_API_WORKERS_MAX_RSS")) if ... else None
```
and `snuba/utils/server.py` passes them to Granian as `workers_lifetime` / `workers_max_rss`. Both are `None` unless the environment supplies them, and `self-hosted`'s compose supplies neither — verified with `docker inspect`, no `SNUBA_API_*` var is present on the container.
These knobs were added by getsentry/snuba#7735, titled *"ref(api): add server settings, allow self-hosted overrides through env"* (Feb 2026) — so they appear to have been built for this deployment, but `self-hosted` never wired them up.
### Suggested fix
In `x-snuba-defaults`, drop the two dead `UWSGI_*` entries and add overridable defaults:
```yaml
SNUBA_API_WORKERS_MAX_RSS: "${SNUBA_API_WORKERS_MAX_RSS:-1024}" # MiB
SNUBA_API_WORKERS_LIFETIME: "${SNUBA_API_WORKERS_LIFETIME:-86400}" # seconds
```
I applied exactly this locally and the worker now starts at ~197 MB with the cap in place. Granian samples RSS every 30s by default and respawns independently of request volume, which matters here: on a low-traffic instance a request-count-based threshold like the old `max-requests: 10000` would rarely be reached anyway.
Worth noting for whoever picks this up: Granian's README warns that *multiple* workers combined with respawn options need kernel >= 5.14 **and** `net.ipv4.tcp_migrate_req` enabled, otherwise queued backlog connections can be dropped during respawn. `SNUBA_API_WORKERS` defaults to 1, so the default path is unaffected — but it argues against raising the worker count as part of any fix here.
I'm not claiming to have located the leak inside Snuba itself; I only have the growth curve. But regardless of where the allocation comes from, self-hosted currently ships with no ceiling on it, and a `mem_limit` on the service would additionally keep an OOM contained to the responsible container instead of taking down a random bystander.
### Event ID
_No response_
Contributor guide
Research direction
Inspect x-snuba-defaults in docker-compose.yml, then read snuba/settings/settings_self_hosted.py and snuba/utils/server.py to trace the worker lifetime and RSS settings into Granian. Use docker inspect or the compose configuration to verify the effective environment. Done means the dead UWSGI_* variables are removed, bounded overridable SNUBA_API_* defaults are present, and the snuba-api worker receives them.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, docker-compose, python
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 70/100