stacklok / stacklok/toolhive

Complete the metrics port move and remove metricsOnTransportPort

Open
#6,384 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
2.2k
Forks
300
Avg merge
1d 15h
Merged PRs (30d)
184

Description

Summary

/metrics is being moved off the transport port onto a dedicated diagnostics port, so
that access to it can be restricted separately from MCP traffic. NetworkPolicy
matches on pods, ports and protocols and cannot filter on HTTP path, so while the two
share a port there is no way to express "allow MCP traffic, deny metrics scraping".

The move is being done in two steps to avoid breaking existing scrape configurations:

  1. Now — both locations served. metricsOnTransportPort defaults to true, so
    /metrics answers on the transport port and the diagnostics port. Nothing breaks.
  2. Later — the cutover. telemetry.DefaultMetricsOnTransportPort flips to false
    and only the diagnostics port serves /metrics.

This issue tracks step 2 and the cleanup after it. The deprecation warning, the docs,
and the metricsOnTransportPort field descriptions all point here for the timeline.

Migration for operators

While both locations are served:

  1. Point your scraper at the diagnostics port. Its resolved address is logged at
    startup — look for prometheus metrics are served on a dedicated diagnostics port, not the application port. It defaults to 9464 but falls back to another port when
    that one is taken, so read the log rather than assuming.
  2. Confirm metrics arrive.
  3. Set metricsOnTransportPort: false (or --otel-metrics-on-transport-port=false) to
    stop serving the old location, and confirm nothing else was still scraping it.

Setting the field explicitly opts out of the eventual default change — an explicit
value is honoured before and after the cutover. Leaving it unset is what picks up the
new default when it lands.

Timeline

The notice window starts when the release containing the deprecation ships, not
when the PRs merge — nothing is visible to users until then.

  • Announcing release: TBD — fill in once cut
  • Target window: ~2 weeks from that release
  • Cutover release: TBD

Releases are currently cut every 4–5 days, so ~2 weeks is roughly 3 releases. Because
merging ships within the week, the cutover PR must be held unmerged until the
window has elapsed rather than parked on main.

Cutover checklist

  • Flip telemetry.DefaultMetricsOnTransportPort to false
  • Release note calling out that scrapers still pointed at the transport port stop
    working, and where to repoint them

Cleanup, after the cutover has shipped

  • Remove telemetry.Config.MetricsOnTransportPort and ServeMetricsOnTransportPort
  • Remove runner.WithMetricsOnTransportPort
  • Remove --otel-metrics-on-transport-port and resolveMetricsOnTransportPort
  • Remove prometheus.metricsOnTransportPort from MCPTelemetryConfig, and move its
    drift-table entry back out of telemetryFieldMappings
  • Remove the dual-serve branches in the runner and vMCP
  • Drop the migration-window sections from docs/observability.md and
    docs/operator/virtualmcpserver-observability.md
  • Regenerate: task docs && task operator-manifests && task crdref-gen

Related

  • #6271 — the umbrella telemetry/diagnostics hardening issue; this is finding E
  • #6296 — moved the proxy half (merged, unreleased)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating telemetry.DefaultMetricsOnTransportPort, MetricsOnTransportPort, ServeMetricsOnTransportPort, and runner.WithMetricsOnTransportPort, then review the runner and vMCP dual-serve branches. Check docs/observability.md and docs/operator/virtualmcpserver-observability.md, and run task docs && task operator-manifests && task crdref-gen after the cleanup. Done means the default is false, deprecated configuration is removed, migration text is gone, and a release note covers the scraper move.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes, prometheus
Domain
documentation, infrastructure, observability
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.