Devolutions / Devolutions/devolutions-gateway

Steady ~15 MiB/day memory growth on an idle gateway (2026.2.4, Linux container)

Open
#1,973 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
159
Forks
28
Avg merge
2d 23h
Merged PRs (30d)
21

Description

### Summary

`devolutions-gateway` 2026.2.4 grows its RSS by a constant ~15.3 MiB/day while completely idle. The growth is linear, does not depend on traffic, and does not level off. In a container with a 256 MiB memory limit this ends in an OOM kill after about 16 days; on a host without a limit it would just keep climbing.

### Environment

- Image: `docker.io/devolutions/devolutions-gateway:2026.2.4` (`sha256:f9ea0f4d…54887`), amd64
- Runtime: Kubernetes 1.34 / containerd, Talos Linux, cgroup v2, memory limit 256 MiB
- Started with the packaged binary directly rather than the pwsh entrypoint:
`/opt/devolutions/gateway/devolutions-gateway --config-path /var/lib/devolutions-gateway`
- Single instance, one HTTP listener, web app enabled with `"Authentication": "None"` (an external reverse proxy does the authentication)

```json
{
"Listeners": [{ "InternalUrl": "http://*:7171", "ExternalUrl": "https://*:443" }],
"WebApp": { "Enabled": true, "Authentication": "None" },
"ProvisionerPublicKeyFile": "…/provisioner.pub",
"ProvisionerPrivateKeyFile": "…/provisioner.key",
"JrlFile": "…/jrl.json",
"TlsVerifyStrict": true
}
```

### Observed behaviour

`container_memory_rss`, one sample per day, over a single 15.7-day process lifetime:

```
08-22 38.4 MiB
08-23 53.5 MiB (+15.1)
08-24 68.6 MiB (+15.1)
08-25 83.8 MiB (+15.2)
08-26 99.1 MiB (+15.3)
08-27 114.5 MiB (+15.4)
08-28 129.9 MiB (+15.4)
08-29 145.3 MiB (+15.4)
08-30 160.5 MiB (+15.2)
08-31 175.8 MiB (+15.3)
09-01 191.6 MiB (+15.8)
09-02 207.7 MiB (+16.1)
09-03 223.5 MiB (+15.9)
09-04 240.1 MiB (+16.5)
09-05 -> OOMKilled at 253.9 MiB against a 256 MiB limit
```

Page cache is not involved: `container_memory_cache` was 0.2 MiB at the time of the kill, so all of this is anonymous memory in the process.

### Why it looks usage-independent

- Between process start and the first user request there were **12 days with zero traffic** — the slope over that stretch is identical to the rest.
- The last log line before the OOM was written **~40 h earlier** (the end of an RDP session). Nothing was logged in those 40 h, and the curve keeps rising at the same rate.
- The replacement process, which nobody has used at all, is leaking at `15.26 MiB/day` — measured as `deriv(container_memory_rss[90m])`.

So something on a timer or on a periodically-repeated request path accumulates memory.

### Hypothesis

The only recurring input to the idle process is the health endpoint: a readiness probe every 20 s and a liveness probe every 60 s, i.e. ~4320 requests/day to `GET /jet/health`. That works out to roughly **3.7 KB retained per health request**, which is in the right ballpark for a per-request leak (a retained span, connection record, or similar).

This is a hypothesis, not a proof — I have not yet run the process with the probes disabled to confirm. One detail that may matter: requests to `/jet/health` do **not** appear in the `devolutions_gateway::middleware::log` output, while every other request does, so they seem to take a different path through the stack.

### Steps to reproduce (expected)

1. Run 2026.2.4 in a container with the config above and no user traffic.
2. Poll `GET /jet/health` every 20 s.
3. Sample RSS hourly for 48 h.

Expected: a straight line at roughly +0.64 MiB/h.

### Questions

- Is a background task (job queue, traffic audit, JRL refresh, mDNS/network-scanner daemon) known to retain memory when idle?
- Is there a build or an env var that would let me hand you a heap profile from the running container?

I can run any diagnostic build you point me at — the instance is easy to redeploy.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the idle process with the Kubernetes probes polling GET /jet/health every 20 seconds and 60 seconds, while sampling container_memory_rss hourly for 48 hours. Compare runs with the probes disabled, then use a diagnostic build or heap profile if available. Done means identifying the retaining path and confirming that RSS no longer grows linearly during an idle run.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, networking, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.