kubernetes-sigs / kubernetes-sigs/controller-runtime
metrics: no way to enable OpenMetrics on the manager's /metrics endpoint
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 3k
- Forks
- 1.3k
- Avg merge
- 2d 5h
- Merged PRs (30d)
- 14
Description
Ran into this trying to expose Prometheus exemplars from a project that uses the manager's metrics server, and I can't find a supported way to do it.
Exemplars only serialise over OpenMetrics. The classic text format has no representation for them, so client_golang drops them at encode time - which means the format the endpoint serves decides whether exemplars work at all, regardless of what the project records into its collectors.
The manager's handler is built with only error handling set, so EnableOpenMetrics stays false:
handler := promhttp.HandlerFor(metrics.Registry, promhttp.HandlerOpts{
ErrorHandling: promhttp.HTTPErrorOnError,
})
(v0.24.1, pkg/metrics/server/server.go)
As far as I can tell there's no way to change that from outside:
- nothing on
metricsserver.Optionsfor handler options, andEnableOpenMetricsdoesn't appear anywhere in the package ExtraHandlersrefuses/metrics- "overriding builtin /metrics endpoint is not allowed" - enforced in bothOptions.ValidateandAddExtraHandlerFilterProvideris the one thing left. the filter gets handed the built handler and can just ignore it, returning its ownpromhttp.HandlerFor(metrics.Registry, promhttp.HandlerOpts{EnableOpenMetrics: true})over the same registry
That last one is what we ended up doing in llm-d (llm-d/llm-d-router#2774). It works, but FilterProvider exists for authn/authz and we're using it to pick a wire format. The part that bothers me most is that a project running with metrics auth disabled now has to carry a filter purely to change the encoding.
Also noticed the TODO(JoelSpeed) on the line that mounts this handler about moving to existing Kubernetes machinery for serving metrics, so this area may already be up for a rework.
What I'd like:
Either an EnableOpenMetrics bool on metricsserver.Options passed through to the internal HandlerOpts, or a HandlerOpts *promhttp.HandlerOpts field so this doesn't need a new field each time promhttp grows one. No strong preference between them.
Shouldn't break anyone either way - promhttp negotiates per request off Accept, so a scraper that doesn't ask for application/openmetrics-text keeps getting the classic format from the same endpoint.
Happy to send the PR if one of those shapes is acceptable.
/kind feature
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in pkg/metrics/server/server.go and trace metricsserver.Options through Options.Validate, AddExtraHandler, and the promhttp.HandlerFor construction. Add a supported option that reaches HandlerOpts without allowing /metrics overrides, then verify that /metrics serves OpenMetrics when requested while retaining the classic format otherwise.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- observability
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 72/100