stacklok / stacklok/toolhive

Remove or rethink the MCP backend headless service

Open
#4,668 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement kubernetes operator
Dominant language
Go
Stars
2.2k
Forks
300
Avg merge
1d 15h
Merged PRs (30d)
184

Description

Problem

The operator/proxyrunner creates a headless service (mcp-<name>-headless) for every SSE/streamable-HTTP MCPServer deployment. This service selects the MCP backend StatefulSet pods directly via app: containerName. There are several problems with this:

1. The headless service bypasses the proxy runner

The service selector points to the raw MCP backend pods, not the proxy runner Deployment. This means anyone in the namespace can resolve pod DNS names and hit the backend directly, skipping all the proxy's auth, session management, and routing logic.

2. The StatefulSet serviceName doesn't match (on main)

On the current main branch, buildStatefulSetSpec sets spec.serviceName to the bare containerName (e.g. "myserver"), but the headless service is named mcp-myserver-headless. These don't match, so Kubernetes never creates the stable per-pod DNS records that the headless service is supposed to enable. The headless service exists but doesn't actually do what it was intended for.

3. For stdio transports, backend pods don't listen on the network

The transportTypeRequiresBackendServices guard skips headless service creation for stdio, but the StatefulSet's spec.serviceName still references a service that won't exist in that case.

4. Nobody consumes the headless service today

The proxy runner uses the ClusterIP service (mcp-<name>) as its target host, set via MCPServiceName in pkg/runtime/setup.go. The headless service is created but nothing reads from it.

Where the code lives

  • Headless service creation: pkg/container/kubernetes/client.go in createHeadlessService() / ensureBackendServices()
  • StatefulSet serviceName: pkg/container/kubernetes/client.go in buildStatefulSetSpec() (line ~450)
  • Service selector: pkg/container/kubernetes/client.go in applyService() (line ~993)
  • Transport guard: pkg/container/kubernetes/client.go in transportTypeRequiresBackendServices()

Proposal

Remove the headless service creation entirely. If per-pod DNS routing is needed in the future for session pinning, it should be designed with clear answers to:

  • How does the proxy discover which pod handled a given session's initialize request (not random selection after the fact)?
  • How do we avoid exposing raw backend pods to the cluster?
  • What happens for stdio backends that don't listen on the network?
  • How do we handle the StatefulSet spec.serviceName immutability constraint for existing deployments?

Context

This came up during review of #4638, which builds on top of the headless service to add per-pod session routing. The foundation needs fixing before we add more plumbing on top.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in pkg/container/kubernetes/client.go with createHeadlessService(), ensureBackendServices(), buildStatefulSetSpec(), applyService(), and transportTypeRequiresBackendServices(). Trace how MCPServiceName is used from pkg/runtime/setup.go and review the deployment implications for existing StatefulSets. Done means the unused headless-service path and its serviceName dependency are removed or redesigned without bypassing the proxy or breaking stdio deployments.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
backend, infrastructure, security
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.