stacklok / stacklok/toolhive

Optimizer mode: allow an operator-pinned set of always-advertised tools

Open
#6,514 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement operator p2 vmcp
Dominant language
Go
Stars
2.2k
Forks
300
Avg merge
1d 15h
Merged PRs (30d)
184

Description

Problem

In optimizer mode the vMCP advertises exactly two tools, find_tool and call_tool, and there is no way for an operator to keep any real tool visible alongside them. #4357 is fixing the description of that pair so agents are more likely to reach for it. This issue is about the case where a better description is not enough and the operator knows, ahead of time, which handful of tools should simply be in the client's tool list.

The list is unconditional:

  • pkg/vmcp/session/optimizerdec/decorator.go:91Tools() returns a copy of optimizerTools, which is OptimizerTools() (:62) and nothing else.
  • pkg/vmcp/server/serve_optimizer.go:81 — the Serve path builds its sdkTools from the same optimizerdec.OptimizerTools(), so both eras behave identically.
  • test/e2e/thv-operator/virtualmcp/virtualmcp_optimizer_composite_test.go:188-201 asserts HaveLen(2) / ConsistOf("find_tool", "call_tool") — composite tools are hidden behind find_tool too, so defining a composite tool is not an escape hatch.
  • ToolOverride (pkg/vmcp/config/config.go:621) is workload-keyed, so it cannot reach the meta-tools either.

Concretely, on a vMCP fronting ~10 read-only kubernetes-mcp-server backends (one per cluster, tool names prefixed with the cluster ID), a request like "list pods in namespace X in cluster Y" names its target unambiguously — but nothing in the advertised pair mentions Kubernetes, pods, or any cluster name, so clients reach for kubectl via bash instead. The server is only used when the user names it explicitly. This is the same failure mode as #4357, from the other side: the useful routing signal is the tool names themselves, and no amount of description text on find_tool reproduces ~135 cluster-prefixed names.

Turning the optimizer off fixes routing and costs the whole catalog (baseline_tokens 36552 here, ~96% reported savings). That is the trade this issue is trying to avoid making: the choice today is all tools or none.

Proposed solution

Let the operator declare a small set of tools that stay advertised in optimizer mode, on top of find_tool / call_tool.

type OptimizerConfig struct {
    // ... existing fields ...

    // AlwaysAdvertise lists backend tool names that stay in the advertised tool
    // list in optimizer mode, alongside find_tool and call_tool. Use it to keep a
    // small, high-traffic subset directly callable so agents route to this server
    // without a find_tool round-trip. Names not present in the session's tool set
    // are ignored. When empty, only the meta-tools are advertised (today's
    // behaviour).
    // +optional
    AlwaysAdvertise []string `json:"alwaysAdvertise,omitempty" yaml:"alwaysAdvertise,omitempty"`
}

Both call sites already have the full advertisable set in hand, so the seam is small:

  • Serve path (serve_optimizer.go): coreTools is already in scope where optimizerFactory is called; filter it by the allowlist and append those definitions to sdkTools, wiring each to its existing core handler rather than to an optimizer meta-handler.
  • Legacy decorator (optimizerdec): NewDecorator takes the allowlist and Tools() returns OptimizerTools() plus the matching subset of d.MultiSession.Tools().

Pinning would only affect advertisement. The optimizer index stays as-is, so a pinned tool is still discoverable through find_tool, and call_tool still resolves it — no behaviour change for clients that ignore the pinned entries.

Notes
  • Admission/authz must run on the pinned entries the same way it runs on the rest, so a pinned tool the caller cannot use is filtered out rather than advertised — cf. #4374.
  • token_metrics.baseline_tokens should probably exclude what is already advertised, otherwise reported savings overstate the win once tools are pinned.
  • A glob or prefix form ("*_namespaces_list") would help fleets where the same tool exists on every backend, but exact names are enough to be useful and are simpler to validate.
Out of scope
  • Per-user or per-group pinning.
  • Automatically choosing what to pin from usage data.
  • Anything about the meta-tool descriptions themselves — that is #4357.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with OptimizerConfig in pkg/vmcp/config/config.go, then read the optimizer call sites in pkg/vmcp/server/serve_optimizer.go and pkg/vmcp/session/optimizerdec/decorator.go. Run test/e2e/thv-operator/virtualmcp/virtualmcp_optimizer_composite_test.go and update coverage so exact pinned tools are advertised alongside the meta-tools, while unavailable or unauthorized tools remain filtered and existing optimizer behavior is preserved.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
api, backend, security, testing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.