Feature request: A way for pods to signal what profiling endpoints are supported
- Dominant language
- Go
- Stars
- 11.7k
- Forks
- 802
- Avg merge
- 1d 19h
- Merged PRs (30d)
- 80
Description
#### Is your feature request related to a problem? Please describe.
We run services written both in Go and Rust, and currently Rust only supports the `/debug/pprof/profile` endpoint. If we just use the same scrape config for all services the agent spams the logs with errors for the rust services because it can't scrape most of the endpoints. It's not a huge issue but it's pretty wasteful to opportunistically scrape everything each interval.
#### Describe the solution you'd like
A set of annotations to be able to override parts of the profiling config would be helpful.
A rough example of how this could look like:
```yaml
pprof.phlare.grafana.com/memory-enabled:
pprof.phlare.grafana.com/block-enabled:
pprof.phlare.grafana.com/goroutine-enabled:
pprof.phlare.grafana.com/mutex-enabled:
pprof.phlare.grafana.com/process-cpu-enabled:
pprof.phlare.grafana.com/fgprof-enabled:
```
#### Describe alternatives you've considered
We have worked around it currently by adding an annotation like `embark.dev/lang` which can be either `golang` or `rust`, and then configure our scrape configs like this:
```jsonnet
local langFilter(lang) = {
action: 'keep',
regex: lang,
source_labels: ['__meta_kubernetes_pod_annotation_embark_dev_lang'],
};
//...
scrape_configs: [
{
job_name: 'kubernetes-pods-golang',
kubernetes_sd_configs: [{ role: 'pod' }],
scrape_interval: '15s',
relabel_configs: relabelConfigs + [langFilter('golang')],
},
{
job_name: 'kubernetes-pods-rust',
kubernetes_sd_configs: [{ role: 'pod' }],
scrape_interval: '15s',
relabel_configs: relabelConfigs + [langFilter('rust')],
profiling_config: {
pprof_config: {
memory: { enabled: false },
block: { enabled: false },
goroutine: { enabled: false },
mutex: { enabled: false },
process_cpu: {
enabled: true,
delta: true,
path: '/debug/pprof/profile',
},
},
},
},
],
```
This is a bit blunt though as it assumes that all services written in a specific language supports the same set of endpoints.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing how profiling_config and pprof_config are processed for Kubernetes pods, including where pod annotations could be read. Define how the proposed endpoint annotations override profiling settings, then verify that unsupported endpoints are not scraped and that the existing Go configuration remains compatible.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes, rust
- Domain
- devops, observability
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100