elastic / elastic/beats

[performance-profiler] Optimize Prometheus info-metric merge in ProcessMetrics hot path

Open
#53,149 1 comment 0 reactions 0 assignees View on GitHub
needs_team
Dominant language
Go
Stars
12.7k
Forks
5k
Avg merge
2d 15m
Merged PRs (30d)
385

Description

## Hot Path
`metricbeat/helper/prometheus/prometheus.go:235-240` performs info-metric merge by iterating all `infoMetrics` and searching matching events via per-label comparisons. On the benchmark path in `metricbeat/helper/prometheus/prometheus_test.go:1249-1279` (`BenchmarkProcessMetricsInfoMerge`), this dominates CPU time for `containers=1000`.

## Profiling Data
**Before:**
```text
go test -run='^$' -bench='BenchmarkProcessMetricsInfoMerge/containers=1000' -benchmem -cpuprofile=/tmp/gh-aw/agent/cpu.before.prof -memprofile=/tmp/gh-aw/agent/mem.before.prof -count=5 ./metricbeat/helper/prometheus

BenchmarkProcessMetricsInfoMerge/containers=1000-4 12 96655038 ns/op 8074962 B/op 101190 allocs/op
BenchmarkProcessMetricsInfoMerge/containers=1000-4 12 99723437 ns/op 8074484 B/op 101189 allocs/op
BenchmarkProcessMetricsInfoMerge/containers=1000-4 12 97631526 ns/op 8074177 B/op 101187 allocs/op
BenchmarkProcessMetricsInfoMerge/containers=1000-4 12 99320253 ns/op 8074390 B/op 101188 allocs/op
BenchmarkProcessMetricsInfoMerge/containers=1000-4 10 100552645 ns/op 8074428 B/op 101188 allocs/op
```

`go tool pprof -top /tmp/gh-aw/agent/cpu.before.prof` (top entries):
```text
flat flat% sum% cum cum%
1250ms 16.09% 16.09% 1480ms 19.05% internal/runtime/maps.(*Map).getWithoutKeySmallFastStr
280ms 3.60% 62.03% 6670ms 85.84% github.com/elastic/beats/v7/metricbeat/helper/prometheus.(*prometheus).ProcessMetrics
240ms 3.09% 68.47% 2310ms 29.73% github.com/elastic/elastic-agent-libs/mapstr.mapFind
```

## Proposed Change
Replace label-by-label event scanning in the info merge path with keyed lookup against `eventsMap`:

- Record precomputed key once when collecting info metrics (`metricbeat/helper/prometheus/prometheus.go:205-208`, `:278-281`).
- Merge info metadata via direct `eventsMap[info.Key]` lookup (`metricbeat/helper/prometheus/prometheus.go:235-239`).
- Keep fallback info-only event grouping keyed by the same cached key (`metricbeat/helper/prometheus/prometheus.go:247-256`).
- Use a fast-path helper for flat map keys to reduce mapstr path parsing overhead (`metricbeat/helper/prometheus/prometheus.go:318-324`).

## Results
**After:**
```text
go test -run='^$' -bench='BenchmarkProcessMetricsInfoMerge/containers=1000' -benchmem -cpuprofile=/tmp/gh-aw/agent/cpu.after.prof -memprofile=/tmp/gh-aw/agent/mem.after.prof -count=5 ./metricbeat/helper/prometheus

BenchmarkProcessMetricsInfoMerge/containers=1000-4 86 14535490 ns/op 8434682 B/op 110189 allocs/op
BenchmarkProcessMetricsInfoMerge/containers=1000-4 85 14933522 ns/op 8434613 B/op 110189 allocs/op
BenchmarkProcessMetricsInfoMerge/containers=1000-4 81 14893223 ns/op 8434680 B/op 110190 allocs/op
BenchmarkProcessMetricsInfoMerge/containers=1000-4 81 14877975 ns/op 8434602 B/op 110189 allocs/op
BenchmarkProcessMetricsInfoMerge/containers=1000-4 82 15052681 ns/op 8434645 B/op 110189 allocs/op
```

`go tool pprof -top /tmp/gh-aw/agent/cpu.after.prof` (top entries):
```text
flat flat% sum% cum cum%
780ms 8.25% 8.25% 1640ms 17.34% runtime.scanObjectsSmall
430ms 4.55% 20.40% 740ms 7.82% compress/flate.(*compressor).deflate
```

**Improvement:** Mean time improved from **98,776,579.8 ns/op** to **14,858,578.2 ns/op** (**84.96% faster**) for `BenchmarkProcessMetricsInfoMerge/containers=1000`.

## Verification
- `go test -count=1 ./metricbeat/helper/prometheus` passed.
- Metric mapping behavior is preserved: merge is keyed by the same `keyLabels.String()` identity already used by `getEvent` (`metricbeat/helper/prometheus/prometheus.go:298-306`).

## Evidence
Commands used:
```text
go test -run='^$' -bench='BenchmarkProcessMetricsInfoMerge/containers=1000' -benchmem -cpuprofile=/tmp/gh-aw/agent/cpu.before.prof -memprofile=/tmp/gh-aw/agent/mem.before.prof -count=5 ./metricbeat/helper/prometheus
go tool pprof -top /tmp/gh-aw/agent/cpu.before.prof
go test -run='^$' -bench='BenchmarkProcessMetricsInfoMerge/containers=1000' -benchmem -cpuprofile=/tmp/gh-aw/agent/cpu.after.prof -memprofile=/tmp/gh-aw/agent/mem.after.prof -count=5 ./metricbeat/helper/prometheus
go tool pprof -top /tmp/gh-aw/agent/cpu.after.prof
go test -count=1 ./metricbeat/helper/prometheus
```

---
[What is this?](https://ela.st/github-ai-tools) | [From workflow: Performance Profiler](https://github.com/elastic/beats/actions/runs/34486501590)

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.
> - [x] expires on Sep 17, 2026, 2:22 PM UTC

Contributor guide

Open the contributing guide

Research direction

Start in metricbeat/helper/prometheus/prometheus.go, especially the info collection, merge, fallback grouping, and map-key helper sections at lines 205-208, 235-256, 278-281, and 318-324. Run BenchmarkProcessMetricsInfoMerge in metricbeat/helper/prometheus/prometheus_test.go, then run go test -count=1 ./metricbeat/helper/prometheus. Done means the tests pass, metric mapping behavior remains unchanged, and the benchmark confirms the reported improvement.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, prometheus
Domain
observability-sre, performance
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.