stacklok / stacklok/toolhive

v0.49.0 vmcp: health probe hangs on the streamable proxy's empty SSE response to server/discover; Ready stuck False

Open
#6,687 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

needs-triage
Dominant language
Go
Stars
2.2k
Forks
300
Avg merge
1d 15h
Merged PRs (30d)
184

Description

After upgrading the operator from v0.8.1 to v0.49.0 (charts via OCI), the vMCP gateway can no longer reliably talk to our two MCPServer backends. They are plain FastAPI/mcp python-SDK servers wrapped stdio→streamable-http by the operator's proxyrunner, and they worked fine against the v0.8.1 gateway (42 tools aggregated, real tool calls).

What v0.49.0's vmcp does, from the backend pod logs and live probing:

  1. Its client sends the new-protocol server/discover (_meta carrying protocol version 2026-07-28). The python SDK answers $-32602 Invalid request parameters (pydantic rejects it as ListTasksRequest/CancelTaskRequest shape errors — the old SDK can't parse the new method).
  2. The health monitor then reports every check failing: legacy initialize step failed: operation timed out: failed to initialize client for backend <name> (timeout): context deadline exceeded — hundreds of consecutive failures at the 30s cadence, circuit breaker never opens.
  3. Meanwhile the backends themselves are instantly healthy: a manual initialize against the proxy URL — modern (2025-03-26) AND legacy (2024-11-05) protocol versions — answers in ~30ms, and tools/list served fine right before.
  4. Net effect on the CR: Ready = False / BackendsUnavailable ("All backends are unhealthy") for hours, phase: Degraded, while the gateway still serves its cached aggregated tool list. Anything gating on the CR's Ready condition (Flux wait: true in our case) wedges permanently.
  5. Separately but probably related: post-initialize client requests to the gateway started returning 404 mid-day (session handling) where the same curl flow worked hours earlier — I did not root-cause this one, flagging it in case it shares the session/dual-era rework.

A pure gateway-image downgrade to v0.8.1 (operator kept at 0.49.0) crash-loops — the old binary won't start against the new generated config — so there's no partial roll-back path; the fleet is stuck with the monitor mis-flagging backends until this is fixed.

Worked around on my side by gating on pod health instead of the CR condition. Happy to provide more logs/traces if useful.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing vmcp's health monitor through the server/discover and legacy initialize paths, then reproduce the empty SSE response against the proxyrunner-wrapped FastAPI backend. Compare the health result with the successful manual initialize and tools/list requests; done means compatible backends are not left unhealthy or Ready=False by this response.

Written by the indexing model from the issue text.

Assessment

Tech stack
fastapi, go, kubernetes, python
Domain
api, backend, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.