kubernetes-sigs / kubernetes-sigs/controller-runtime

metrics: no way to enable OpenMetrics on the manager's /metrics endpoint

Open
#3,592 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

kind/feature
Dominant language
Go
Stars
3k
Forks
1.3k
Avg merge
2d 5h
Merged PRs (30d)
14

Description

Ran into this trying to expose Prometheus exemplars from a project that uses the manager's metrics server, and I can't find a supported way to do it.

Exemplars only serialise over OpenMetrics. The classic text format has no representation for them, so client_golang drops them at encode time - which means the format the endpoint serves decides whether exemplars work at all, regardless of what the project records into its collectors.

The manager's handler is built with only error handling set, so EnableOpenMetrics stays false:

handler := promhttp.HandlerFor(metrics.Registry, promhttp.HandlerOpts{
    ErrorHandling: promhttp.HTTPErrorOnError,
})

(v0.24.1, pkg/metrics/server/server.go)

As far as I can tell there's no way to change that from outside:

  • nothing on metricsserver.Options for handler options, and EnableOpenMetrics doesn't appear anywhere in the package
  • ExtraHandlers refuses /metrics - "overriding builtin /metrics endpoint is not allowed" - enforced in both Options.Validate and AddExtraHandler
  • FilterProvider is the one thing left. the filter gets handed the built handler and can just ignore it, returning its own promhttp.HandlerFor(metrics.Registry, promhttp.HandlerOpts{EnableOpenMetrics: true}) over the same registry

That last one is what we ended up doing in llm-d (llm-d/llm-d-router#2774). It works, but FilterProvider exists for authn/authz and we're using it to pick a wire format. The part that bothers me most is that a project running with metrics auth disabled now has to carry a filter purely to change the encoding.

Also noticed the TODO(JoelSpeed) on the line that mounts this handler about moving to existing Kubernetes machinery for serving metrics, so this area may already be up for a rework.

What I'd like:

Either an EnableOpenMetrics bool on metricsserver.Options passed through to the internal HandlerOpts, or a HandlerOpts *promhttp.HandlerOpts field so this doesn't need a new field each time promhttp grows one. No strong preference between them.

Shouldn't break anyone either way - promhttp negotiates per request off Accept, so a scraper that doesn't ask for application/openmetrics-text keeps getting the classic format from the same endpoint.

Happy to send the PR if one of those shapes is acceptable.

/kind feature

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in pkg/metrics/server/server.go and trace metricsserver.Options through Options.Validate, AddExtraHandler, and the promhttp.HandlerFor construction. Add a supported option that reaches HandlerOpts without allowing /metrics overrides, then verify that /metrics serves OpenMetrics when requested while retaining the classic format otherwise.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
observability
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.