grafana / grafana/pyroscope

Feature request: A way for pods to signal what profiling endpoints are supported

Open
#2,080 0 comments 3 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
11.7k
Forks
802
Avg merge
1d 19h
Merged PRs (30d)
80

Description

#### Is your feature request related to a problem? Please describe.

We run services written both in Go and Rust, and currently Rust only supports the `/debug/pprof/profile` endpoint. If we just use the same scrape config for all services the agent spams the logs with errors for the rust services because it can't scrape most of the endpoints. It's not a huge issue but it's pretty wasteful to opportunistically scrape everything each interval.

#### Describe the solution you'd like

A set of annotations to be able to override parts of the profiling config would be helpful.

A rough example of how this could look like:

```yaml
pprof.phlare.grafana.com/memory-enabled:
pprof.phlare.grafana.com/block-enabled:
pprof.phlare.grafana.com/goroutine-enabled:
pprof.phlare.grafana.com/mutex-enabled:
pprof.phlare.grafana.com/process-cpu-enabled:
pprof.phlare.grafana.com/fgprof-enabled:
```

#### Describe alternatives you've considered

We have worked around it currently by adding an annotation like `embark.dev/lang` which can be either `golang` or `rust`, and then configure our scrape configs like this:

```jsonnet
local langFilter(lang) = {
action: 'keep',
regex: lang,
source_labels: ['__meta_kubernetes_pod_annotation_embark_dev_lang'],
};
//...
scrape_configs: [
{
job_name: 'kubernetes-pods-golang',
kubernetes_sd_configs: [{ role: 'pod' }],
scrape_interval: '15s',
relabel_configs: relabelConfigs + [langFilter('golang')],
},
{
job_name: 'kubernetes-pods-rust',
kubernetes_sd_configs: [{ role: 'pod' }],
scrape_interval: '15s',
relabel_configs: relabelConfigs + [langFilter('rust')],
profiling_config: {
pprof_config: {
memory: { enabled: false },
block: { enabled: false },
goroutine: { enabled: false },
mutex: { enabled: false },
process_cpu: {
enabled: true,
delta: true,
path: '/debug/pprof/profile',
},
},
},
},
],
```

This is a bit blunt though as it assumes that all services written in a specific language supports the same set of endpoints.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing how profiling_config and pprof_config are processed for Kubernetes pods, including where pod annotations could be read. Define how the proposed endpoint annotations override profiling settings, then verify that unsupported endpoints are not scraped and that the existing Go configuration remains compatible.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes, rust
Domain
devops, observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.