Devolutions / Devolutions/devolutions-gateway
Steady ~15 MiB/day memory growth on an idle gateway (2026.2.4, Linux container)
- Dominant language
- Rust
- Stars
- 159
- Forks
- 28
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 21
Description
### Summary
`devolutions-gateway` 2026.2.4 grows its RSS by a constant ~15.3 MiB/day while completely idle. The growth is linear, does not depend on traffic, and does not level off. In a container with a 256 MiB memory limit this ends in an OOM kill after about 16 days; on a host without a limit it would just keep climbing.
### Environment
- Image: `docker.io/devolutions/devolutions-gateway:2026.2.4` (`sha256:f9ea0f4d…54887`), amd64
- Runtime: Kubernetes 1.34 / containerd, Talos Linux, cgroup v2, memory limit 256 MiB
- Started with the packaged binary directly rather than the pwsh entrypoint:
`/opt/devolutions/gateway/devolutions-gateway --config-path /var/lib/devolutions-gateway`
- Single instance, one HTTP listener, web app enabled with `"Authentication": "None"` (an external reverse proxy does the authentication)
```json
{
"Listeners": [{ "InternalUrl": "http://*:7171", "ExternalUrl": "https://*:443" }],
"WebApp": { "Enabled": true, "Authentication": "None" },
"ProvisionerPublicKeyFile": "…/provisioner.pub",
"ProvisionerPrivateKeyFile": "…/provisioner.key",
"JrlFile": "…/jrl.json",
"TlsVerifyStrict": true
}
```
### Observed behaviour
`container_memory_rss`, one sample per day, over a single 15.7-day process lifetime:
```
08-22 38.4 MiB
08-23 53.5 MiB (+15.1)
08-24 68.6 MiB (+15.1)
08-25 83.8 MiB (+15.2)
08-26 99.1 MiB (+15.3)
08-27 114.5 MiB (+15.4)
08-28 129.9 MiB (+15.4)
08-29 145.3 MiB (+15.4)
08-30 160.5 MiB (+15.2)
08-31 175.8 MiB (+15.3)
09-01 191.6 MiB (+15.8)
09-02 207.7 MiB (+16.1)
09-03 223.5 MiB (+15.9)
09-04 240.1 MiB (+16.5)
09-05 -> OOMKilled at 253.9 MiB against a 256 MiB limit
```
Page cache is not involved: `container_memory_cache` was 0.2 MiB at the time of the kill, so all of this is anonymous memory in the process.
### Why it looks usage-independent
- Between process start and the first user request there were **12 days with zero traffic** — the slope over that stretch is identical to the rest.
- The last log line before the OOM was written **~40 h earlier** (the end of an RDP session). Nothing was logged in those 40 h, and the curve keeps rising at the same rate.
- The replacement process, which nobody has used at all, is leaking at `15.26 MiB/day` — measured as `deriv(container_memory_rss[90m])`.
So something on a timer or on a periodically-repeated request path accumulates memory.
### Hypothesis
The only recurring input to the idle process is the health endpoint: a readiness probe every 20 s and a liveness probe every 60 s, i.e. ~4320 requests/day to `GET /jet/health`. That works out to roughly **3.7 KB retained per health request**, which is in the right ballpark for a per-request leak (a retained span, connection record, or similar).
This is a hypothesis, not a proof — I have not yet run the process with the probes disabled to confirm. One detail that may matter: requests to `/jet/health` do **not** appear in the `devolutions_gateway::middleware::log` output, while every other request does, so they seem to take a different path through the stack.
### Steps to reproduce (expected)
1. Run 2026.2.4 in a container with the config above and no user traffic.
2. Poll `GET /jet/health` every 20 s.
3. Sample RSS hourly for 48 h.
Expected: a straight line at roughly +0.64 MiB/h.
### Questions
- Is a background task (job queue, traffic audit, JRL refresh, mDNS/network-scanner daemon) known to retain memory when idle?
- Is there a build or an env var that would let me hand you a heap profile from the running container?
I can run any diagnostic build you point me at — the instance is easy to redeploy.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the idle process with the Kubernetes probes polling GET /jet/health every 20 seconds and 60 seconds, while sampling container_memory_rss hourly for 48 hours. Compare runs with the probes disabled, then use a diagnostic build or heap profile if available. Done means identifying the retaining path and confirming that RSS no longer grows linearly during an idle run.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend, networking, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100