[performance-profiler] Optimize Prometheus info-metric merge in ProcessMetrics hot path
- Dominant language
- Go
- Stars
- 12.7k
- Forks
- 5k
- Avg merge
- 2d 15m
- Merged PRs (30d)
- 385
Description
## Hot Path
`metricbeat/helper/prometheus/prometheus.go:235-240` performs info-metric merge by iterating all `infoMetrics` and searching matching events via per-label comparisons. On the benchmark path in `metricbeat/helper/prometheus/prometheus_test.go:1249-1279` (`BenchmarkProcessMetricsInfoMerge`), this dominates CPU time for `containers=1000`.
## Profiling Data
**Before:**
```text
go test -run='^$' -bench='BenchmarkProcessMetricsInfoMerge/containers=1000' -benchmem -cpuprofile=/tmp/gh-aw/agent/cpu.before.prof -memprofile=/tmp/gh-aw/agent/mem.before.prof -count=5 ./metricbeat/helper/prometheus
BenchmarkProcessMetricsInfoMerge/containers=1000-4 12 96655038 ns/op 8074962 B/op 101190 allocs/op
BenchmarkProcessMetricsInfoMerge/containers=1000-4 12 99723437 ns/op 8074484 B/op 101189 allocs/op
BenchmarkProcessMetricsInfoMerge/containers=1000-4 12 97631526 ns/op 8074177 B/op 101187 allocs/op
BenchmarkProcessMetricsInfoMerge/containers=1000-4 12 99320253 ns/op 8074390 B/op 101188 allocs/op
BenchmarkProcessMetricsInfoMerge/containers=1000-4 10 100552645 ns/op 8074428 B/op 101188 allocs/op
```
`go tool pprof -top /tmp/gh-aw/agent/cpu.before.prof` (top entries):
```text
flat flat% sum% cum cum%
1250ms 16.09% 16.09% 1480ms 19.05% internal/runtime/maps.(*Map).getWithoutKeySmallFastStr
280ms 3.60% 62.03% 6670ms 85.84% github.com/elastic/beats/v7/metricbeat/helper/prometheus.(*prometheus).ProcessMetrics
240ms 3.09% 68.47% 2310ms 29.73% github.com/elastic/elastic-agent-libs/mapstr.mapFind
```
## Proposed Change
Replace label-by-label event scanning in the info merge path with keyed lookup against `eventsMap`:
- Record precomputed key once when collecting info metrics (`metricbeat/helper/prometheus/prometheus.go:205-208`, `:278-281`).
- Merge info metadata via direct `eventsMap[info.Key]` lookup (`metricbeat/helper/prometheus/prometheus.go:235-239`).
- Keep fallback info-only event grouping keyed by the same cached key (`metricbeat/helper/prometheus/prometheus.go:247-256`).
- Use a fast-path helper for flat map keys to reduce mapstr path parsing overhead (`metricbeat/helper/prometheus/prometheus.go:318-324`).
## Results
**After:**
```text
go test -run='^$' -bench='BenchmarkProcessMetricsInfoMerge/containers=1000' -benchmem -cpuprofile=/tmp/gh-aw/agent/cpu.after.prof -memprofile=/tmp/gh-aw/agent/mem.after.prof -count=5 ./metricbeat/helper/prometheus
BenchmarkProcessMetricsInfoMerge/containers=1000-4 86 14535490 ns/op 8434682 B/op 110189 allocs/op
BenchmarkProcessMetricsInfoMerge/containers=1000-4 85 14933522 ns/op 8434613 B/op 110189 allocs/op
BenchmarkProcessMetricsInfoMerge/containers=1000-4 81 14893223 ns/op 8434680 B/op 110190 allocs/op
BenchmarkProcessMetricsInfoMerge/containers=1000-4 81 14877975 ns/op 8434602 B/op 110189 allocs/op
BenchmarkProcessMetricsInfoMerge/containers=1000-4 82 15052681 ns/op 8434645 B/op 110189 allocs/op
```
`go tool pprof -top /tmp/gh-aw/agent/cpu.after.prof` (top entries):
```text
flat flat% sum% cum cum%
780ms 8.25% 8.25% 1640ms 17.34% runtime.scanObjectsSmall
430ms 4.55% 20.40% 740ms 7.82% compress/flate.(*compressor).deflate
```
**Improvement:** Mean time improved from **98,776,579.8 ns/op** to **14,858,578.2 ns/op** (**84.96% faster**) for `BenchmarkProcessMetricsInfoMerge/containers=1000`.
## Verification
- `go test -count=1 ./metricbeat/helper/prometheus` passed.
- Metric mapping behavior is preserved: merge is keyed by the same `keyLabels.String()` identity already used by `getEvent` (`metricbeat/helper/prometheus/prometheus.go:298-306`).
## Evidence
Commands used:
```text
go test -run='^$' -bench='BenchmarkProcessMetricsInfoMerge/containers=1000' -benchmem -cpuprofile=/tmp/gh-aw/agent/cpu.before.prof -memprofile=/tmp/gh-aw/agent/mem.before.prof -count=5 ./metricbeat/helper/prometheus
go tool pprof -top /tmp/gh-aw/agent/cpu.before.prof
go test -run='^$' -bench='BenchmarkProcessMetricsInfoMerge/containers=1000' -benchmem -cpuprofile=/tmp/gh-aw/agent/cpu.after.prof -memprofile=/tmp/gh-aw/agent/mem.after.prof -count=5 ./metricbeat/helper/prometheus
go tool pprof -top /tmp/gh-aw/agent/cpu.after.prof
go test -count=1 ./metricbeat/helper/prometheus
```
---
[What is this?](https://ela.st/github-ai-tools) | [From workflow: Performance Profiler](https://github.com/elastic/beats/actions/runs/34486501590)
Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.
> - [x] expires on Sep 17, 2026, 2:22 PM UTC
Contributor guide
Research direction
Start in metricbeat/helper/prometheus/prometheus.go, especially the info collection, merge, fallback grouping, and map-key helper sections at lines 205-208, 235-256, 278-281, and 318-324. Run BenchmarkProcessMetricsInfoMerge in metricbeat/helper/prometheus/prometheus_test.go, then run go test -count=1 ./metricbeat/helper/prometheus. Done means the tests pass, metric mapping behavior remains unchanged, and the benchmark confirms the reported improvement.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, prometheus
- Domain
- observability-sre, performance
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 72/100