vmcp: all VirtualMCPServer instances grow 2-3x in memory over ~24h regardless of client activity
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 2.2k
- Forks
- 300
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 184
Description
Bug description
All VirtualMCPServer (vmcp) instances we run grow 2-3x in RSS memory over
roughly 24 hours, independent of client connection activity, eventually
threatening node memory headroom (this drove a node-wide OOM incident for
us). A prior working theory - that this was client-side reconnect churn -
is falsified by continued growth measured well after fixing that churn
cause on the client side.
Steps to reproduce
- Deploy 2+ VirtualMCPServer instances with different backend counts (we
ran 4, with 12/13/14/19 backends respectively). - Measure RSS via
kubectl top podat two points ~23h apart, with the
deployments not restarted and no client-driven reconnect storms in
between. - Compare growth against the backend MCPServer/MCPRemoteProxy Deployments
in the same namespace (control group) over the same window.
Expected behavior
Memory should stabilize/plateau. The control-group backend Deployments
show exactly this behavior (flat over the same 23h window).
Actual behavior
| instance | backends | RSS T+0h | RSS T+23h | growth | rate |
|---|---|---|---|---|---|
| A | 19 | 476Mi | 1476Mi | 3.1x | ~43 Mi/h |
| B | 14 | 349Mi | 683Mi | 2.0x | ~14.5 Mi/h |
| C | 13 | 184Mi | 446Mi | 2.4x | ~11.4 Mi/h |
| D | 12 | 172Mi | 343Mi | 2.0x | ~7.4 Mi/h |
Growth rate scales roughly with backend count, consistent with a
per-backend resource retained per some recurring operation (health-check
tick, session-scoped init, etc.) rather than one fixed leak. A clean pod
restart drops each instance back to a ~35-100Mi baseline immediately.
Control group (same namespace, same 23h window, unrestarted): three
backend proxy Deployments stayed flat (264Mi / 249Mi / 175Mi at both
measurement points).
Environment
- ToolHive operator + vmcp: v0.34.0
- Single-node Kubernetes (Talos), 4 independent VirtualMCPServer instances
Additional context
- Related but distinct: #4062 (closed) fixed a monotonic memory leak in
thv-proxyrunnerunder streamable-http session churn - that's the
individual-backend-proxy component. Our observation is in the
vmcp/VirtualMCPServer aggregator process itself; our control-group
backend proxy pods stayed flat over the same window, so this does not
look like the same code path re-surfacing. - We have not isolated which internal resource is growing (no profiling
access in this deployment). Happy to gather a pprof heap profile across
a multi-hour window if vmcp exposes one, or to test against specific
config knobs if pointed at likely candidates.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The report names no files, tests, or entry points; start by locating the VirtualMCPServer implementation and its recurring per-backend operations. Reproduce the multi-hour RSS comparison and collect a pprof heap profile if vmcp exposes one. Done means identifying the retained resource and demonstrating that memory stabilizes without client reconnect activity.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- backend, infrastructure, observability
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100