prometheus-community / prometheus-community/stackdriver_exporter
Unbounded concurrent Monitoring API requests exhaust memory and OOM-kill the container
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 295
- Forks
- 117
- Avg merge
- 9d 14h
- Merged PRs (30d)
- 7
Description
Root cause confirmed: each scrape fetches time series for every metric
descriptor of every configured project concurrently, with no limit — one
goroutine + HTTP request + JSON decode per descriptor, per project. When
google.projects.filter (or a long google.project-ids list) resolves to
many projects, this spawns far more concurrent Monitoring API requests than
a memory-constrained pod can service at once, leading to OOM kills. This is
worse in containers specifically, since GOMAXPROCS reports the host's
vCPUs rather than the container's cgroup CPU limit — far more goroutines get
scheduled than the pod's actual quota, and CPU throttling stalls many of
them mid-decode with their buffers unfreed.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No file, test, or entry point is named in the issue. Trace the scrape path that fetches time series for each metric descriptor and project, then inspect how goroutines, HTTP responses, and JSON buffers are managed; the issue is complete only when bounded concurrency prevents the described container OOM condition, with tests or measurements defining that behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, google-cloud
- Domain
- backend, cloud, observability-sre
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100