stacklok / stacklok/toolhive

vmcp: all VirtualMCPServer instances grow 2-3x in memory over ~24h regardless of client activity

Open
#5,860 4 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

needs_more_info
Dominant language
Go
Stars
2.2k
Forks
300
Avg merge
1d 15h
Merged PRs (30d)
184

Description

Bug description

All VirtualMCPServer (vmcp) instances we run grow 2-3x in RSS memory over
roughly 24 hours, independent of client connection activity, eventually
threatening node memory headroom (this drove a node-wide OOM incident for
us). A prior working theory - that this was client-side reconnect churn -
is falsified by continued growth measured well after fixing that churn
cause on the client side.

Steps to reproduce

  1. Deploy 2+ VirtualMCPServer instances with different backend counts (we
    ran 4, with 12/13/14/19 backends respectively).
  2. Measure RSS via kubectl top pod at two points ~23h apart, with the
    deployments not restarted and no client-driven reconnect storms in
    between.
  3. Compare growth against the backend MCPServer/MCPRemoteProxy Deployments
    in the same namespace (control group) over the same window.

Expected behavior

Memory should stabilize/plateau. The control-group backend Deployments
show exactly this behavior (flat over the same 23h window).

Actual behavior

instance backends RSS T+0h RSS T+23h growth rate
A 19 476Mi 1476Mi 3.1x ~43 Mi/h
B 14 349Mi 683Mi 2.0x ~14.5 Mi/h
C 13 184Mi 446Mi 2.4x ~11.4 Mi/h
D 12 172Mi 343Mi 2.0x ~7.4 Mi/h

Growth rate scales roughly with backend count, consistent with a
per-backend resource retained per some recurring operation (health-check
tick, session-scoped init, etc.) rather than one fixed leak. A clean pod
restart drops each instance back to a ~35-100Mi baseline immediately.

Control group (same namespace, same 23h window, unrestarted): three
backend proxy Deployments stayed flat (264Mi / 249Mi / 175Mi at both
measurement points).

Environment

  • ToolHive operator + vmcp: v0.34.0
  • Single-node Kubernetes (Talos), 4 independent VirtualMCPServer instances

Additional context

  • Related but distinct: #4062 (closed) fixed a monotonic memory leak in
    thv-proxyrunner under streamable-http session churn - that's the
    individual-backend-proxy component. Our observation is in the
    vmcp/VirtualMCPServer aggregator process itself; our control-group
    backend proxy pods stayed flat over the same window, so this does not
    look like the same code path re-surfacing.
  • We have not isolated which internal resource is growing (no profiling
    access in this deployment). Happy to gather a pprof heap profile across
    a multi-hour window if vmcp exposes one, or to test against specific
    config knobs if pointed at likely candidates.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The report names no files, tests, or entry points; start by locating the VirtualMCPServer implementation and its recurring per-backend operations. Reproduce the multi-hour RSS comparison and collect a pprof heap profile if vmcp exposes one. Done means identifying the retained resource and demonstrating that memory stabilizes without client reconnect activity.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
backend, infrastructure, observability
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.